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COMPTROLLER  GENERAL  OF  THE  UNITED  STATES 
WASHINGTON  O.C.  20641 


To  the  President  of  the  Senate  and  the 
Speaker  of  the  House  of  Representatives 

The  Congress  is  increasingly  concerned  about  basic  research  and  its 
importance  in  making  technological  advancements,  improving  productivity, 
and  finding  cures  for  dreaded  diseases  such  as  cancer.  Much  of  this 
country's  nondefense  basic  research  is  funded  through  basic  research  grants 
made  by  the  National  Science  Foundation  and  the  National  Institutes  of 
Health  to  colleges,  universities,  and  other  nonprofit  research  institutions. 
This  report  assesses  the  systems  used  by  these  agencies  to  determine  those 
research  proposals  that  are  to  be  funded  and  how  scientific  performance  on 
the  grants  is  assessed  when  continued  support  is  provided. 

We  made  this  review  because  in  recent  years  several  congressional 
comnittees  and  Members  of  the  Congress  have  expressed  concern  over  whether 
the  Federal  Government's  basic  research  grants  are  adequately  evaluated  and 
whether  only  the  most  deserving  of  all  re-.aarch  proposals  are  supported. 

In  this  report,  we  focus  on  the  main  element  of  the  scientific  performance 
accountability  system— called  peer  review— which  is  used  in  large  measure 
to  evaluate  proposed  as  well  as  completed  research.  The  report  includes 
recommendations  for  Improvements  to  make  the  systems  more  effective. 

We  are  sending  copies  of  the  report  to  appropriate  House  and  Senate 
committees.  Representatives  and  Senators  who  have  a  particular  interest  in 
the  subject,  the  Secretary  of  Health  and  Human  Services,  the  Directors  of 
the  Office  of  Management  and  Budget,  the  Office  of  Science  and  Technology 
Policy,  the  National  Institutes  of  Health,  and  the  National  Science  Founda¬ 
tion,  and  to  other  Interested  parties.  We  will  also  make  copies  available 
to  interested  organizations  and  individuals,  as  appropriate,  on  request. 


!  ,v.-4 


COMPTROLLER  GENERAL’S 
REPORT  TO  THE  CONGRESS 


BETTER  ACCOUNTABILITY 
PROCEDURES  NEEDED  IN 
NSF  AND  NIH  RESEARCH 
GRANT  SYSTEMS 


DIGEST 

The  quality  of  the  scientific  performance 
accountability  system  is  of  great  importance 
to  the  Congress,  the  Federal  Government,  and 
university  researchers.  Peer  review,  in  which 
\  selected  researchers  called  "peer  reviewers" 
evaluate  the  scientific  merit  of  research  pro¬ 
posals,  is  recognized  as  the  primary  component 
of  this  system.  The  Congress  has  been  concerned 
with  whether  taxpayers'  dollars  are  being  in¬ 
vested  wisely  in  these  grants.  One  commonly 
asked  question  by  the  Congress  and  congressional 
committees  with  jurisdiction  over  Federal  agen¬ 
cies  funding  basic  research  at  universities 
(such  as  NSF  and  NIH)  is  how  and  to  what  extent 
basic  research  is  being  evaluated.  Another 
question  is  whether  the  systems  the  agencies 
use  to  determine  from  all  research  proposals 
those  most  deserving  of  support  work  properly. 

This  review  was  made  to  find  out  how  the  NSF 
and  NIH  scientific  performance  accountability 
systems  are  working  and  to  identify  improvements 
that  could  make  both  systems  more  effective. 

In  fiscal  year  1980,  the  National  Science 
Foundation  (NSF)  and  the  National  Institutes 
of  Health  (NIH)  provided  75  percent  of  the 
total  amount  awarded  by  the  Federal  Government, 
or  $2.2  billion,  for  the  support  of  basic  re¬ 
search  at  colleges  and  universities.  The  bulk 
of  both  agencies'  research  support  is  pro¬ 
vided  through  individual  research  grants. 

OBJECTIVES,  SCOPE,  AND  METHODOLOGY 

GAO  selected  a  random  sample  of  75  NSF  and  NIH 
basic  research  grants  (whose  funding  ended  dur¬ 
ing  fiscal  year  1978)  made  to  6  major  research 
universities  ranked  among  the  top  20  in  Federal 
funds  received.  GAO  had  two  broad  objectives 
in  examining  the  scientific  performance  accounta¬ 
bility  systems  NSF  and  NIH  use  (NIH  peer  review 
group,  NSF  panel,  and  NSF  ad  hoc)  to  review  these 
grants.  The  first  objective  was  to  determine 
whether  research  was  being  funded  but  not  accom¬ 
plished  and  if  grants  were  subsequently  getting 
renewed  support  in  spite  of  poor  performance. 

The  second  objective  was  to  assess  how  well  the 
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scientific  accountability  process  identified 
unproductive  researchers  and  prevented  them 
from  receiving  continued  funding.  To  achieve 
these  objectives,  GAO  examined  the  five  main 
elements  of  the  accountability  process  (proposal 
submission,  peer  review,  award  decision,  monitor¬ 
ing  the  research,  and  evaluating  the  research) 
as  the  agencies  intended  them  to  operate, 
then  examined  the  steps  in  the  process  as  they 
operate  in  practice.  GAO  did  not  attempt  to 
make  scientific  judgments  regarding  any  aspect 
of  the  grants  it  reviewed. 

RESEARCH  GRANT  SCIENTIFIC 
PERFORMANCE  ACCOUNTABILITY  SYSTEMS 
OF  NSF  AND  NIH  CAN  BE  IMPROVED 

Although  the  scientific  performance  accounta¬ 
bility  systems  are  basically  the  same  at  NSF 
and  NIH,  the  procedures  differ  significantly. 

GAO  believes  that  some  of  the  NIH  peer  review 
procedures  have  certain  advantages  over  those 
at  NSF.  (See  chapter  2.) 

For  instance,  at  NIH,  of  the  25  research  grants 
reviewed,  researchers  on  23  sought  additional 
funding  from  NIH  to  continue  their  research. 

The  peer  reviewers  prevented  4,  or  17  percent, 
of  these  from  getting  funded  because  of  lack 
of  progress,  impact  of  the  research  already 
done,  or  for  other  reasons.  Also,  at  NIH, 
peer  reviewers'  comments  directly  affected  7 
of  the  19  that  received  continued  funding  by 
eliminating  some  proposed  research  objectives 
that  lacked  merit,  or  by  reducing  the  funding 
or  time  requested  to  do  the  proposed  research. 

At  NSF,  of  the  50  research  grants  reviewed, 
none  of  the  27  researchers  who  sought  continued 
funding  had  their  requests  turned  down,  but  the 
peer  reviewers'  comments  did  play  a  role  in  10 
cases  in  that  some  research  objectives  were 
eliminated  or  the  funding  or  time  requested  to 
do  the  research  was  reduced. 

NSF  does  not  ask  peer  reviewers  to  comment  on 
the  performance  of  the  immediately  preceding 
grant  when  the  researcher  submits  a  proposal 
for  a  renewal  grant.  The  reviewers  are  asked 
to  evaluate  the  scientific  merit  of  the  re¬ 
newal  proposal  and  the  researcher's  overall 
track  record.  Of  the  50  NSF  grants  GAO  re¬ 
viewed,  27  (54  percent)  were  renewed  to  con¬ 
tinue  the  same  line  of  research.  In  only  6 
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of  the  27  cases  (22  percent)  did  GAO  find 
any  evaluative  comments  in  the  peer  review 
renewal  proposal  critiques  concerning  a  re¬ 
searcher's  performance  during  the  preceding 
grant.  (See  p.  21.) 

In  contrast,  at  NIH,  where  peer  reviewers  are 
asked  when  reviewing  renewal  proposals  to  com¬ 
ment  on  the  immediately  preceding  grant,  GAO 
found  that  in  95  percent  of  the  cases  (18  of 
19  grants  renewed),  the  researcher's  perfor¬ 
mance  on  the  preceding  grant  was  evaluated  by 
the  peers  in  critiquing  the  renewal  proposal. 

(See  p.  22. ) 

Even  if  peer  reviewers  were  asked  to  comment  on 
the  immediately  preceding  grant,  NSF  renewal 
proposal  instructions  do  not  require  sufficient 
information  to  insure  that  such  an  evaluation 
is  feasible.  NSF,  unlike  NIH,  does  not  require 
an  identification  of  the  preceding  grant's 
objectives  and  progress  toward  their  accomplish¬ 
ment  or  identification  of  the  preceding  grant's 
publications  in  renewal  proposals. 

(See  p.  22. ) 

Renewal  proposals  which  easily  identify  preced¬ 
ing  grant  results  would  be  particularly  impor¬ 
tant  at  NSF  because  (1)  they  often  would  be  the 
only  source  of  these  results  at  the  time  of 
renewal;  and  (2)  different  peer  reviewers  and 
program  officers  usually  review  successive 
grants. 

For  new  project  proposals,  neither  NSF  nor  NIH 
requires  researchers  to  discuss  the  prior  grant 
results  or  identify  prior  grant  publications. 
Also,  peer  reviewers  are  not  asked  to  evaluate 
scientific  progress  on  the  prior  grant.  This 
is  particularly  important  when  researchers  have 
two  or  more  grants  at  the  time  they  submit  new 
project  proposals,  because  there  is  little  way 
of  determining  which  grant  produced  the  research 
results  or  publications  that  might  be  discussed 
in  the  proposal.  These  procedures  do  not  pre¬ 
clude  researchers  from  avoiding  accountability 
by  continually  proposing  new  projects.  Twenty 
percent  of  the  researchers  in  GAO's  review  re¬ 
ceived  grants  for  new  projects.  (See  pp.  27-28.) 

NIH  automatically  forwards  peer  review  comments 
to  researchers.  NSF  forwards  them  only  when 
the  researcher  requests  them.  NSF's  policy  may 
inhibit  some  researchers  from  receiving  use¬ 
ful  information.  In  some  instances,  this 
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policy  has  an  adverse  effect  on  the  conduct  of 
the  research.  GAO  found  that  about  two-thirds 
of  the  researchers  did  not  receive  the  peer 
review  comments.  Further,  for  many  NSF  grants 
which  are  panel-reviewed,  no  summary  of  the  panel 
deliberations  was  available  to  be  forwarded 
to  the  researcher.  (See  pp.  33-34.) 

Neither  NSF  nor  NIH  uniformly  monitors  the 
progress  or  evaluates  the  results  of  research 
grants.  The  tools  available  to  perform  both  of 
these  functions,  progress  reports,  final  reports, 
and  publications,  are  used  by  program  officers 
in  a  variety  of  ways.  Some  program  officers 
rely  on  peer  review  of  subsequent  proposals  to 
identify  unproductive  researchers  and  determine 
whether  the  researcher’s  prior  work  justifies 
continued  support.  However,  19  percent  of 
the  researchers  in  the  GAO  sample  did  not  seek 
another  grant  from  NSF  or  NIH.  (See  pp.  35-38.) 

Universities  traditionally  do  not  monitor  the 
progress  or  evaluate  the  results  of  research 
grants.  Their  role  is  limited  to  reviewing 
proposals  prepared  by  researchers  for  adminis¬ 
trative  matters  and  adherence  to  university 
policy  prior  to  submission  to  NSF  or  NIH. 

(See  p.  39. ) 

RESEARCHERS  ARE  ACCOUNTABLE 
FOR  RESULTS— NOT  ACCOMPLISHING 
OBJECTIVES 

Most  of  the  researchers  who  were  awarded  re¬ 
newal  grants  did  not  accomplish  all  of  the 
objectives  of  the  immediately  preceding  grant. 
Peer  reviewers  and  program  officers  were  not 
concerned  by  this  because  they  believe  results 
are  more  important  than  accomplishing  the  pro¬ 
posed  objectives,  and  because,  for  the  most 
part,  they  do  not  expect  all  objectives  to  be 
accomplished.  Most  of  them,  however,  did 
expect  the  researchers  to  attempt  the  grant's 
objectives.  GAO  believes  that  this  method  of 
operation  is  basically  sound.  However,  unless 
the  renewal  proposal  identifies  the  preceding 
grant's  objectives,  it  is  difficult  to  deter¬ 
mine  if  the  objectives  were  attempted.  NSF, 
unlike  NIH,  does  not  require  renewal  proposals 
to  restate  the  preceding  grant’s  objectives. 

(See  pp.  45-47. ) 

NIH  requires  researchers  to  specifically  state 
the  objectives  (specific  aims)  to  be  attempted 
during  the  grant  period  as  well  as  the  overall 
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objective  of  the  line  of  research.  NSF  does 
not  distinguish  between  these  types  of  objec¬ 
tives.  (See  p.  47. ) 

Neither  NSF  nor  NIH  specifies  the  extent  to 
which  researchers  can  deviate  from  a  grant's 
original  objectives  without  prior  agency  ap¬ 
proval.  (See  p.  49.) 

RECOMMENDATIONS 

GAO  recommends  that  the  Director  of  NSF  require 
that : 

— Renewal  proposal  progress  reports  identify 
the  objectives,  evidence  of  progress  toward 
their  achievement,  any  major  changes  in 
direction  or  emphasis  and  the  rationale  for 
such  changes,  publications,  and/or  other  out¬ 
put  from  a  researcher's  immediately  preceding 
grant. 

— Peer  reviewers  be  asked  when  reviewing  renewal 
proposals  to  specifically  comment  on  a  re¬ 
searcher's  performance  on  the  immediately 
preceding  grant. 

— The  documentation  of  panel  peer  review  delib¬ 
erations  include  the  major  elements  required 
of  the  NIH  peer  review  group  summary  statement 
when  individual  peer  reviewers’  written  re¬ 
views  do  not  provide  this  information. 

— Peer  review  comments  be  automatically  sent 
to  researchers. 

— Proposals  identify  the  research  objectives 
to  be  undertaken  during  the  grant  period. 

GAO  recommends  that  the  directors  of  NSF  and  NIH 
require  that: 

— Proposals  for  new  projects  include  evidence 
of  progress  from  the  prior  grant(s). 

— Peer  reviewers  be  furnished  any  available 
final  technical  reports  and  listings  of  pub¬ 
lications  from  the  prior  grant(s)  when 
researchers  seek  funding  for  new  projects. 

— More  systematic  and  uniform  review  of  annual 
progress  reports  be  made  by  the  program 
officers. 
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— More  specific  guidelines  be  established  re¬ 
garding  the  extent  to  which  researchers  can 
change  grant  objectives  without  prior  agency 
approval . 

AGENCY  COMMENTS 

NSF  and  NIH's  parent  organization,  the  Depart¬ 
ment  of  Health  and  Human  Services,  generally 
concurred  with  GAO's  recommendations  and  with 
one  exception  by  NSF  agreed  to  examine  current 
practices  and/or  develop  better  guidelines  to 
implement  GAO's  recommendations.  HHS  stated  that 
the  report  fairly  presents  the  issues  involved. 
NSF  noted  that  while  improvements  should  always 
be  sought  in  any  system,  any  changes  must  be 
considered  in  the  context  of  workload  implica¬ 
tions.  The  recommendation  NSF  took  exception 
to  has  been  revised  to  reflect  NSF's  views. 

MATTERS  FOR  CONSIDERATION 
BY  THE  CONGRESS 


The  House  Committee  on  Government  Operations 
and  the  Senate  Committee  on  Governmental  Affairs, 
during  oversight  hearings  on  NSF,  NIH,  or  on 
matters  relating  to  universities  and  research 
grant  accountability,  should  consider  the  effec¬ 
tiveness  of  the  scientific  performance  accounta¬ 
bility  systems  at  NIH  and  especially  at  NSF.  Also, 
the  House  Committees  on  Science  and  Technology 
and  Appropriations  and  the  Subcommittee  on  Ap¬ 
propriations — HUD  and  Independent  Agencies,  Senate 
Committee  on  Appropriations,  during  their  fiscal 
year  1983  budget  hearings,  should  consider  NSF 
and  NIH  actions  taken  to  improve  the  research 
grant  scientific  performance  accountability  sys¬ 
tems  since  the  hearings  mentioned  on  pages  6-7. 
These  systems  determine  the  quality  of  much  of 
the  basic  research  conducted  at  the  Nation's  uni¬ 
versities,  and  this  research  is  vital  to  the 
Nation's  welfare.  The  systems  NSF  and  NIH  use 
should  work  as  effectively  as  possible,  espe¬ 
cially  considering  shrinking  research  budgets 
and  the  ever-increasing  demand  for  technological 
advances.  The  recommendations  made  in  this 
report  will  help  improve  the  scientific  perform¬ 
ance  accountability  systems  at  NIH  and  especially 
at  NSF. 
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INTRODUCTION 

Research  and  development  are  vital  to  our  national  economy. 
Economic  growth,  national  security,  and  quality  of  life  all  de¬ 
pend  critically  on  technological  development.  Research  contrib¬ 
utes  to  the  prestige  and  leadership  of  our  Nation  in  international 
affairs.  Scientific  research  leads  to  improvements  in  agricul¬ 
ture,  in  the  diagnosis  and  treatment  of  disease,  and  in  techno¬ 
logies  which  increase  productivity  and  lead  to  new  products.  The 
keystone  of  and  the  underpinning  for  technological  development  is 
basic  research,  important  because  of  broad  potential  for  social 
benefits,  not  by  its  ability  to  generate  specific  products  or 
services.  Basic  research  is  inherently  exploratory.  Furthermore, 
the  ultimate  significance  of  results  may  not  be  visible  for  years. 

BACKGROUND 


The  objective  of  basic  research  is  to  produce  new  knowledge 
without  regard  to  its  application.  Basic  research  is  extremely 
important  in  providing  the  fundamental  knowledge  necessary  for 
progress.  Basic  research  inherently  involves  a  long-term  view, 
and  provides  seed  for  broad  social  benefits. 

Project  grant  funding  began  its  development  in  various  pri¬ 
vate  foundations  before  World  War  II.  The  Federal  Government 
adopted  the  process  as  its  primary  mechanism  to  support  basic  re¬ 
search  in  colleges  and  universities  because  it  avoided  detailed 
and  short-term  political  control  of  research.  For  the  last  3 
decades,  the  Federal  Government  has  been  the  primary  supporter  of 
basic  research  in  universities.  Over  50  percent  of  the  Nation's 
basic  research  is  now  performed  in  colleges  and  universities. 

Federal  agencies  provided  about  $2.9  billion  to  universities 
and  colleges  during  fiscal  year  (FY)  1980  to  conduct  basic  re¬ 
search,  which  is  about  54  percent  of  the  total  Federal  funds  spent 
on  basic  research.  Federal  Government  funds  which  universities 
receive  for  basic  research  are  about  72  percent  of  the  total  funds 
universities  get  for  basic  research.  Over  $2.2  billion,  or  75 
percent  of  the  $2.9  billion  provided  by  the  Federal  Government  to 
universities  came  from  two  agencies — the  National  Science  Founda¬ 
tion  ( NSF )  and  the  National  Institutes  of  Health  (NIH) . 

THE  NATIONAL  SCIENCE  FOUNDATION 

The  National  Science  Foundation  (NSF)  is  an  independent 
Federal  agency  established  under  the  National  Science  Foundation 
Act  of  1950,  as  amended  (42  U.S.C.  1861  et  seq. )  1970.  Its  pri¬ 
mary  mission  is  to  strengthen  U.S.  science  by  supporting  basic 
research  and  science  education.  NSF  is  the  principal  Federal 
agency  that  supports  non-mission  oriented  basic  research  at  uni¬ 
versities  and  colleges.  In  some  fields,  NSF  provides  the  dominant 
share  of  Federal  basic  research  support:  more  than  69  percent 
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in  ground-based  astronomy,  60  percent  in  environmental  sciences 
and  over  50  percent  in  mathematics  and  engineering.  About  2,000 
colleges,  universities,  and  other  institutions  participate  in 
NSF  programs.  Of  the  26,000  proposals  reviewed  in  1980  by  NSF , 
11,500  (44  percent)  were  funded,  for  a  total  of  $653  million. 

NSF  determines  the  quality  of  research  proposals  through  a 
system  of  peer  review.  The  scientists  and  engineers  who  partici¬ 
pate  in  the  peer  review  process  give  their  views  and  evaluations 
of  proposed  projects.  The  Foundation  has  six  organizational 
units,  called  directorates,  which  operate  the  proposal  evaluation 
process  and  award  research  grants.  Each  directorate  is  headed 
by  an  assistant  director  of  the  Foundation  and  is  subdivided  into 
divisions,  sections,  and  programs  representing  specific  areas  of 
science.  The  key  individual  in  each  program  is  the  program 
officer  who  manages  the  program's  proposal  evaluation  process.  A 
program  officer  is  the  focal  point  between  NSF  and  the  science 
community,  being  responsible  for  recommending  whether  a  proposal 
should  be  funded. 

THE  NATIONAL  INSTITUTES  OF  HEALTH 

The  National  Institutes  of  Health  (NIH)  is  one  of  the  six 
agencies  composing  the  U.S.  Public  Health  Service.  NIH  is  under 
the  leadership  of  a  Director  who  reports  to  both  the  Assistant 
Secretary  for  Health  and  the  Surgeon  General  of  the  Department  of 
Health  and  Human  Services.  NIH  is  one  of  the  world's  foremost 
prestigious  biomedical  research  centers,  and  is  the  focal  point 
for  Federal  mission  oriented  biomedical  basic  research  and 
research  support.  Hie  mission  of  NIH  is  to  improve  the  health  of 
the  people  of  the  United  States.  To  accomplish  this  mission,  NIH 
conducts  and  supports  research  about  the  cause,  diagnosis,  preven¬ 
tion,  and  cure  of  diseases  of  man,  the  processes  of  human  growth 
and  development,  the  biological  effects  of  environmental  contami¬ 
nants,  and  the  sciences  related  to  health.  NIH  is  composed  of  11 
separate  research  Institutes,  each  of  which  supports  biomedical 
research  programs,  and  a  division  of  research  resources.  All  but 
one  Institute  are  located  at  the  NIH  complex  in  Bethesda,  Maryland. 
The  other  is  located  at  Research  Triangle  Park,  North  Carolina. 

NIH  is  by  far  the  largest  single  source  of  funds  for  bio¬ 
medical  research  conducted  in  the  Nation's  universities  and 
medical  schools.  Its  FY  1980  budget  for  basic  research  grants 
was  nearly  $1.6  billion.  Table  1  illustrates  the  size  of  and 
compares  NSF's  and  NIH's  proposal  review/grant  award  activity. 
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Table  1 


Comparing  NSF  and  NIH  Proposal  Review  and  Grant  Award  Activity 


Proposals  reviewed  a/ 

NSF 

NIH 

FY  1979 

25,000 

26,000 

FY  1980 

26,000 

26,600 

Grants  awarded 

FY  1979 

10,700 

15,300 

FY  1980 

11,500 

16,500 

Grant  funds  awarded 

FY  1979 

$575  million 

$1,400  million 

FY  1980 

$653  million 

$1,600  million 

Average  cost  of  a  grant  (1980) 

$56,800 

$97,000 

Average  length  of  a  grant  (1980) 

2.2  years 

3.2  years 

a /  Includes  proposals  for  new  or  competing  grants  and  proposals 
to  continue  existing  or  noncompeting  grants. 

RESEARCH  GRANTS  ARE  USED 
FOR  BASIC  RESEARCH 


Federal  agencies  use  three  basic  types  of  funding  mechanisms 
to  support  basic  research  in  colleges  and  universities:  research 
grants,  research  contracts,  or  cooperative  agreements.  In  gene¬ 
ral,  grants  are  used  when  the  principal  purpose  is  to  support  or 
stimulate  research  and  the  Federal  agency  is  not  substantially 
involved  in  the  management  or  performance  of  the  activity.  Con¬ 
tracts  are  used  when  the  Federal  agency  is  obtaining  specific 
types  of  information,  products,  or  services.  Usually,  contracts 
are  negotiated,  managed,  and  monitored  by  Federal  agencies  much 
more  closely  than  grants.  Cooperative  agreements,  as  introduced 
pursuant  to  the  Federal  Grant  and  Cooperative  Agreement  Act  of 
1977,  are  similar  to  grants  in  that  they  are  oriented  toward 
support  or  stimulation  of  a  particular  activity  but  provide  for 
varying  degrees  of  involvement  by  the  Federal  agency.  Although 
both  NSF  and  NIH  use  research  grants  and  contracts  almost  exclu¬ 
sively,  the  bulk  of  research  support  they  provide  to  universities 
is  through  individual  research  project  grants. 

Most  proposals  for  grant  support  originate  with  individual 
researchers  in  a  college  or  university  who  develop  a  proposed  plan 
for  research  within  an  area  of  interest  to  them  and  to  the  agency 
from  which  support  is  solicited.  Although  grants  are  usually 
awarded  to  an  organization  such  as  a  college  or  university  that 
sponsors  and  conducts  basic  research,  the  agencies  are  still 
accountable  to  the  public  for  the  funds  used  to  finance  basic 
research  grants. 
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MAINTAINING  ACCOUNTABILITY 
WITHOUT  INHIBITING  CREATIVITY 


The  health  of  science  and  technology,  and  particularly  the 
state  of  basic  research  in  this  country,  are  subjects  of  much 
concern.  Our  international  leadership  in  science  and  technology 
is  being  challenged,  and  national  policy  issues  such  as  those  in¬ 
volved  in  energy  resource  development  and  environmental  protection 
increasingly  involve  science  and  technology.  Also,  the  current 
pressures  of  budgetary  constraints  have  made  accountability  a 
leading  issue  in  all  areas  of  Government  spending.  Thus,  the  Con¬ 
gress  and  the  public  are  concerned  with  how  to  obtain  adequate 
accountability  of  federally  supported  basic  research  grants  while, 
at  the  same  time,  not  unduly  inhibiting  researchers'  creativeness. 

Each  university  is  independent,  and  research  is  performed  in 
independent  departments,  composed  of  individual,  autonomous  re¬ 
searchers.  The  structure  of  this  environment  is  generally  nonhi- 
erarchical  and  tends  to  be  loose  and  flexible.  The  keystone  of 
the  research  process  is  the  individual  researcher  or  the  small 
group  of  researchers  who  perform  the  work.  The  process  of  inves¬ 
tigation  itself,  like  the  overall  "climate,"  is  characterized  by 
a  lack  of  hierarchy.  The  researcher  conceives,  directs,  performs, 
and  publishes  the  work,  often  in  conjunction  with  graduate  stu¬ 
dents,  who  are  essentially  practicing  apprentices.  The  researcher 
has  a  heightened  sense  of  self-reliance,  which  serves  as  crucial 
motivation  for  the  work.  In  fact,  a  researcher's  independence  has 
come  to  be  viewed  by  many  scientists,  as  well  as  nonscientists, 
as  necessary  to  scientific  excellence. 

The  distinction  between  basic  and  applied  research  is  not 
always  clear,  but  it  is  generally  recognized  that  basic  research 
is  characterized  by  certain  attributes  which  make  oversight  and 
accountability  difficult.  Basic  research  is  inherently  long- 
range  and  it  is  more  difficult  to  plan  definitively  than  is 
applied  research.  Potential  payoffs  are  highly  uncertain,  with 
little  or  no  assurance  of  positive  results,  even  when  the  scien¬ 
tific  methods  used  are  correct.  During  the  course  of  a  basic 
research  investigation,  it  is  frequently  desirable  to  change  di¬ 
rection  or  methodology  to  overcome  difficulties  and  take  advantage 
of  opportunities. 

Notwithstanding  this  uncertainty,  however,  a  good  research 
proposal  for  a  grant  award  should  describe  the  phenomenon  to  be 
studied  or  the  scientific  problem  for  which  a  solution  is  sought. 
In  the  context  of  the  existing  state  of  knowledge,  the  scientific 
method  or  approach  the  researcher  plans  to  use  can  be  described. 
Hie  proposal  should  include  enough  information  about  the  intended 
objectives  and  scientific  approach,  not  only  to  justify  the  merits 
of  the  proposal,  but  also  to  provide  a  basis  for  assessing  the  re¬ 
search  performance  in  accordance  with  the  commitment  stated  or 
implied  in  the  proposal.  Progress  on  a  first-time  grant  in  a  new 
area  may  be  minimal  and  not  achieve  significant  advances  or  re¬ 
sults  prior  to  requesting  a  grant  renewal,  but  it  should  be 
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possible  to  determine  whether  the  research  was  performed  in 
accordance  with  the  scope  and  direction  indicated  in  the  original 
proposal  and  whether  sufficient  progress  had  been  made  to  warrant 
continued  Federal  funding. 

There  are  two  principal  forms  of  accountability:  financial 
and  administrative,  which  focuses  on  evidence  of  financial  pro¬ 
priety  and  compliance  with  administrative  requirements;  and  sci¬ 
entific,  which  focuses  on  scientific  performance.  In  funding 
basic  research,  financial  and  administrative  accountability 
refers  to  the  degree  to  which  funds  are  spent  within  the  terms 
of  the  research  agreement  without  diversion,  fraud,  or  waste. 

This  accountability  is  enforced  largely  by  the  Federal  Government. 

Scientific  accountability  is  concerned  with  the  quality  of 
performance  and  the  scientific  integrity  of  the  research  in  ac¬ 
cordance  with  the  standards  and  protocol  of  the  scientific  commu¬ 
nity  and  in  relation  to  the  commitment  made  or  implied  in  the 
proposal  which  won  the  grant  award.  The  scientific  community 
outside  the  Government  plays  a  major  role  in  self  governing  in 
scientific  accountability. 

In  the  research  community,  selected  researchers  called  peer 
reviewers  play  a  major  role  in  deciding  what  work  will  be  sup¬ 
ported,  who  shall  carry  out  the  work,  and  what  is  significant. 

Peer  review  is  the  primary  system  for  evaluating  research  propo¬ 
sals  and  searching  out  opportunities  to- advance  science.  It  is 
also  essentially  a  method  of  accounting  for  and  reviewing  research 
on  its  own  terms.  Peer  review  is  the  method  by  which  the  Federal 
Government  assures  itself  and  the  public  of  the  quality  and  sig¬ 
nificance  of  the  basic  research  being  supported  through  the  grant 
process.  Thus,  it  is  an  essential  element  in  the  accountability 
system. 

THE  CONGRESS  IS  INTERESTED  IN 
HOW  WELL  PEER  REVIEW  IS  WORKING 

The  quality  of  the  scientific  performance  accountability 
system--which  is  mainly  peer  review— is  of  great  importance  to 
the  Congress,  the  Federal  Government,  and  university  researchers. 
Peer  review  is  recognized  by  these  groups  as  the  primary  source 
of  scientific  accountability  for  the  public  funds  awarded  to 
researchers  in  basic  research  grants.  However,  the  widespread 
acceptance  of  peer  review  to  ensure  that  scientific  research  is 
funded  fairly  and  effectively  and  to  provide  scientific  accounta¬ 
bility  has  been  questioned.  The  Congress,  for  example,  has  been 
concerned  with  whether  taxpayers'  dollars  are  being  invested 
wisely  in  basic  research  grants.  Criticisms  of  peer  review  made 
by  the  Congress  often  center  on  the  secrecy  of  the  systems,  in¬ 
adequate  consideration  of  past  performance  of  a  researcher  in 
evaluating  the  scientific  potential  of  a  research  proposal,  and 
inadequate  protection  against  possible  abuses. 
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Concern  for  the  quality  of  peer  review  has  grown  because 
the  Congress  has  increased  its  interest  in  scientific  research 
for  improving  technological  development  and  productivity,  and 
because  the  Government  needs  to  be  assured  that  it  is  getting 
value  for  its  research  investment  because  of  budget  constraints. 
One  commonly  asked  question  by  the  Congress  and  congressional 
committees  with  jurisdiction  over  Federal  agencies  funding  basic 
research  at  universities  (namely,  NSF  and  NIH),  is  how  and  to 
what  extent  basic  research  is  being  evaluated.  Another  question 
is  whether  the  systems  the  agencies  use  to  determine  from  all 
research  proposals  those  most  deserving  of  support  work  properly 
and  provide  adequate  safeguards  against  abuse. 

The  fundamental  issue  in  supporting  basic  research  is  how  to 
choose  from  all  researchers'  proposals  those  most  likely  to  pro¬ 
duce  high  quality  research  that  will  advance  basic  knowledge. 

The  peer  review  system  removes  scientific  accountability  from  the 
public  domain.  Researchers--who  are  the  peer  reviewers — largely 
determine  who  gets  funded,  how  well  they  perform,  and  the  quality 
of  the  research  results.  Thus,  the  scientific  accountability 
system  is  "closed"  in  that  review  by  non-researchers  is  not  done 
in  most  cases.  The  Congress  requires  that  this  "closed"  system 
works  well  and  that  the  best  research  is  supported  with  the  funds 
available.  Considering  the  scale  and  importance  of  the  scien¬ 
tific  performance  accountability  system,  the  Congress*  concern  is 
warranted. 

This  report  focuses  on  the  scientific  performance  accounta¬ 
bility  systems  used  by  NSF  and  NIH  because  these  two  agencies 
provide  75  percent  of  all  Federal  basic  research  grant  funds  that 
go  to  the  Nation's  colleges  and  universities.  Also,  both  agencies 
rely  on  peer  review  as  the  primary  means  of  assuring  scientific 
accountability  over  basic  research  grants. 

During  the  past  several  years  the  Congress,  and  particularly 
the  authorization  and  appropriations  committees  with  jurisdiction 
over  NSF  and  NIH,  have  held  special  oversight  hearings,  conducted 
investigations,  or  have  focused,  during  the  annual  budget  hear¬ 
ings,  on  the  scientific  performance  accountability  systems  used 
by  NSF  and  NIH.  In  July  1975,  the  Subcommittee  on  Science, 
Research,  and  Technology  of  the  House  Committee  on  Science  and 
Technology  conducted  6  days  of  special  oversight  hearings  on  how 
NSF  makes  individual  grant  awards.  The  Subcommittee's  hearings 
were  prompted  by  (1)  congressional  and  public  concern  that  NSF 
might  be  supporting  questionable  research,  (2)  the  spreading  be¬ 
lief  that  Government  operations  ought  to  be  open  to  public  scru¬ 
tiny  (since  at  that  time  peer  review  of  research  proposals  was 
conducted  in  almost  total  secrecy),  and  (3)  concern  that  tightness 
of  funds  might  result  in  complaints  from  the  scientific  community 
about  NSF '8  decisionmaking  systems. 

The  Subcommittee  on  HUD-Independent  Agencies,  Senate  Commit¬ 
tee  on  Appropriations,  during  hearings  on  NSF's  fiscal  year  1979 
and  1980  budget  requests,  focused  on  how  NSF  evaluates  research 
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results  of  the  immediately  preceding  project  grant  before  provid¬ 
ing  more  funds  to  continue  the  project.  The  Subcommittee  ques¬ 
tioned  how  NSF  determined  that  a  research  project  was  worthwhile, 
if  the  funds  used  on  a  previous  project  were  well  spent,  and 
whether  NSF  ever  looked  at  the  results  of  previous  projects  before 
providing  additional  funding  to  continue  a  research  project. 

In  August  1979,  the  House  Committee  on  Appropriations  di¬ 
rected  its  Surveys  and  Investigations  staff  to  examine  the  proc¬ 
ess  NIH  uses  to  review  and  approve  researcher-initiated  research 
project  grants  for  use  during  its  fiscal  year  1981  NIH  budget 
hearings.  The  Committee  asked,  "Given  a  finite  amount  of  Federal 
funds  that  can  be  allocated  to  medical  research,  how  can  they 
best  be  divided  among  competing  demands?" 

In  April  1981,  the  newly  constituted  Investigations  and  Over¬ 
sight  Subcommittee  of  the  House  Committee  on  Science  and  Technol¬ 
ogy  held  hearings  on  the  veracity  of  the  scientific  publications 
process  as  an  accurate  measure  of  the  quality  of  research  results 
produced  under  federally-sponsored  research  grants.  The  hearings 
stemmed  from  selected  cases  and  widespread  allegations  of  fraud 
in  the  scientific  publications  process. 

The  sensitivity  of  and  relative  secrecy  with  which  peer  re¬ 
view  is  conducted  surrounds  the  controversy  and  importance  of 
peer  review  in  providing  scientific  accountability  over  basic 
research  grants.  Sponsoring  agencies,  and  the  scientists  who  are 
peer  reviewers,  believe  and  steadfastly  support  the  notion  that 
anonymity  of  individual  peer  reviewers'  comments  must  be  main¬ 
tained  if  honest  and  objective  peer  review  evaluations  are  to  be 
made  on  basic  research  proposals. 

As  a  result,  on  the  one  hand  the  Congress  needs  to  be  assured 
that  public  funds  are  well  spent  on  basic  research  grants,  which 
entails  thorough  scrutiny  of  the  systems  used  to  award  the  funds. 
On  the  other  hand,  the  sensitivity  of  the  "closed"  nature  of  the 
peer  review  system  needs  to  be  recognized,  as  it  determines  who 
gets  the  funds.  Thus  peer  review  represents  a  "double-edged 
sword."  Because  of  this  dichotomy  and  the  intense  interest  the 
Congress  and  congressional  committees  have  shown  in  peer  review, 
this  report  examines  the  scientific  performance  accountability 
systems  NSF  and  NIH  use  for  basic  research  grants,  placing  special 
emphasis  on  peer  review,  and  shows  how  well  the  systems  are  work¬ 
ing  and  the  improvements  needed  to  make  them  work  better. 

OBJECTIVES,  SCOPE,  AND  METHODOLOGY 


Our  review  of  the  systems  NSF  and  NIH  use  to  assure  the 
scientific  accountability  of  basic  research  granting  procedures 
had  two  broad  objectives.  One  was  to  determine  from  a  randomly 
selected  sample  of  75  NSF  and  NIH  research  grants  whether  re¬ 
search  was  being  funded  but  not  being  accomplished  and  if  the 
grants  were  subsequently  getting  renewed  support  in  spite  of  poor 
performance.  By  poor  performance,  we  mean  that  condition  in 
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which  a  researcher  did  not  attempt  the  intended  research  objec¬ 
tives,  did  not  make  reasonable  progress  towards  those  objectives, 
or  did  not  produce  identifiable  research  results  1/  without  any 
valid  reasons  as  to  why;  and/or  the  quality  of  the  research  work 
that  was  done  was  below  the  level  where  continuing  the  work  would 
not  have  been  in  the  best  interests  of  advancing  science  with  the 
limited  funds  available.  The  other  objective  was  to  assess  how 
well  the  scientific  performance  accountability  process  identified 
unproductive  researchers  and  prevented  them  from  receiving  con¬ 
tinued  funding.  We  also  reviewed  NSF's  and  NIH's  grant  monitoring 
and  evaluation  systems  which  check  on  grants  after  they  are  made. 

The  information  concerning  our  sample  grants  in  this  report 
is  representative  of  most  basic  research  grants  made  to  major 
research  universities  that  are  evaluated  by  the  peer  review 
systems  NSF  and  NIH  use.  As  a  result,  we  believe  our  evaluation 
results  generally  reflect  the  status  of  the  scientific  performance 
accountability  process  used  for  similar  NSF  and  NIH  basic  research 
grants . 

We  selected  a  stratified,  random  sample  of  NSF  and  NIH  basic 
research  grants  which  ended  during  fiscal  year  1978  made  to  6 
major  research  universities  ranked  among  the  top  20  in  terms  of 
Federal  research  funds  received.  The  schools  selected  were  MIT, 
Yale,  University  of  Chicago,  University  of  Wisconsin  at  Madison, 
Stanford,  and  the  University  of  California  at  Berkeley.  These 
universities  are  geographically  dispersed  and  include  public  and 
private  institutions.  We  limited  our  selection  of  universities 
to  those  in  the  top  20  because  (1)  they  get  40  percent  of  all 
Federal  research  grant  funds,  and  (2)  we  believe  our  results  and 
recommendations  would  have  greater  credibility  in  the  scientific 
community  if  they  were  based  on  grants  involving  prestigious 
research  universities.  We  limited  our  selection  of  grants  to 
those  which  NSF  and  NIH  had  identified  as  basic  research  grants. 
This  type  of  grant  represents  the  bulk  of  these  agencies'  research 
support.  Also,  scientific  accountability  is  hardest  to  assess 
on  basic  research  grants. 

The  subjects  covered  in  the  grants  included  chemistry, 
physics,  materials  research,  engineering,  geology,  oceanography, 
the  life  and  biological  sciences,  ecology,  mathematics,  computer 
science,  law,  economics,  and  other  social  sciences.  In  reviewing 
grants  in  these  areas,  NSF  and  NIH  use  three  somewhat  different 
peer  review  systems.  NIH  uses  one  system  which  is  fairly  uniform 
throughout  NIH.  NSF,  however,  uses  two  different  systems — ad  hoc 


^/Research  grant  results  are  evidence  of  progress  toward  objec- 
~  tives  and  the  output  describing  such  progress  including  publi¬ 
cations,  completed  manuscripts  or  other  printed  material  such 
as  papers  delivered  at  symposia  or  the  progress  report  sections 
of  renewal  proposals.  Such  results  are  a  major  element  in 
evaluating  performance  under  the  grant. 
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and  panel.  Since  we  wanted  to  compare  the  three  systems,  the 
total  number  of  grants  made  to  the  six  universities  in  our  sample 
were  grouped  or  stratified  according  to  the  peer  review  system 
used  to  evaluate  them.  NIH  had  364  grants,  the  NSF  ad  hoc,  322; 
and  the  NSF  panel,  181. 

Our  original  sampling  plan  was  to  select  25  grants  from  each 
of  the  three  systems.  However,  the  actual  sample  ended  up  with 
25  for  NIH,  29  for  NSF  ad  hoc,  and  21  for  NSF  panel.  The  NSF 
sanples  changed  slightly  because  of  errors  in  the  data  NSF  pro¬ 
vided  which  did  not  become  known  until  after  the  sample  was 
selected.  Sanple  sizes  were  derived  based  on  statistical  pre¬ 
cision  at  the  90  percent  confidence  level  with  an  error  rate  of 
+13  percent.  Hie  change  in  NSF  sample  sizes  does  not  materially 
affect  the  samples'  statistical  precision. 

Our  evaluation  approach  to  accomplish  the  review  objectives 
included: 

— Reviewing  the  files  of  the  75  sample  grants  to  identify 
the  research  objectives  proposed  by  the  researcher,  and 
the  peer  reviewers'  and  agency  program  officials'  comments 
regarding  the  proposed  research.  We  also  reviewed,  for  . 
many,  the  grant  which  preceded  the  sample  grant,  and  all 
of  the  renewal  applications  for  continued  funding  that 
V  immediately  followed  the  sample  grants.  As  a  result,  we 
reviewed  over  150  grants  and/or  grant  proposals  to  under¬ 
stand  more  fully  what  resulted  from  the  sample  grants  and 
to  evaluate  better  the  scientific  performance  account¬ 
ability  process. 

--Identifying  the  publications  or  other  identifiable  results 
that  came  from  the  grants  and  tracing,  where  possible,  the 
published  results  to  grant  objectives. 

— Discussing  with  the  individual  researchers  the  sample 

grants'  objectives,  performance,  and  results,  to  learn  how 
many  were  accomplished  or  why  objectives  were  not  accom¬ 
plished,  and  to  verify  the  publications  or  other  identifi¬ 
able  material  that  resulted. 

— Discussing  69  of  the  grants  with  agency  program  officials 
responsible  for  approving  and  monitoring  the  grants.  Re¬ 
sponsible  program  officials  knowledgeable  about  the  remain¬ 
ing  six  grants  were  no  longer  at  the  agencies  at  the  time 
of  our  review. 

--Contacting  58  peer  reviewers  on  16  sample  grant  renewals 
to  determine  whether  the  reviewers  had  sufficient  knowl¬ 
edge  of  the  preceding  grants  to  provide  reasonably  in¬ 
formed  judgments  regarding  the  success  of  the  sample  grant 
to  justify  continued  funding.  The  16  grants  selected  were 
chosen  because  analysis  of  the  sample  grants  showed  that 
some  research  objectives  were  not  attempted,  few  or  no 
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published  results  appeared,  or  other  problems  indicated 
that  less  than  satisfactory  performance  was  made  under 
the  sample  grant  that  peer  reviewers  of  the  renewals 
should  have  been  aware  of  in  deciding  whether  renewed 
funding  was  warranted. 

— Examining  NSF  and  NIH  policies  and  procedures  for  evalu¬ 
ating  basic  research  proposals,  operating  the  peer  review 
systems,  and  monitoring  an>*  evaluating  research  grant 
results . 

— Reviewing  several  reports  and  studies  of  scientific 
accountability  over  *  isic  research  grants  and  the  peer 
review  systems  at  NSF  and  NIH. 

We  accomplished  the^s  objectives  by  reviewing  our  sample  of 
research  grants  to  determine  what  was  done,  the  results  produced 
under  the  sample  grants,  and  how  requests  for  renewed  support  by 
the  researchers  with  the  sample  grants  fared  in  the  accountability 
(peer  review)  systems.  We  compared  the  work  that  was  done  and 
the  results  produced  under  the  sample  grants  to  the  information 
that  was  available  on  the  decisions  made  to  provide  continued 
funding  under  the  renewal  grants  to  see  if  the  accountability 
systems  had  considered  the  same  information  we  found. 

Our  tests  were  based  on  NSF's  and  NIH’s  premise  that  poor 
performance  under  a  preceding  grant  (as  defined  on  pages  7-8) 
should  in  most  cases  result  in  disapproval  of  renewed  funding  on 
a  renewal  request.  This  premise  was  emphasized  in  testimony  by 
NSF  officials  before  a  Senate  Appropriations  Subcommittee  in 
response  to  Subcommittee  members'  questions  concerning  how 
research  grant  results  are  evaluated. 

Another  premise  we  used  concerned  how  the  quality  of  research 
results  is  determined.  According  to  NSF  and  NIH  officials  and 
most  researchers  we  contacted,  research  grant  results  need  to  be 
published  in  so-called  refereed  or  peer  reviewed  scientific  jour¬ 
nals  to  achieve  maximum  credibility.  Peer  reviewed  publications 
are  the  hallmark  of  a  successful  research  grant,  and  in  large 
measure  determine  the  "track  record"  of  the  researcher.  This  cri¬ 
terion  is  a  primary  means  to  judge  how  successful  each  grant  is. 

Evaluation  approach  limitations 

We  did  not  attempt  to  make  scientific  judgments  regarding 
any  aspect  of  the  grants  we  reviewed.  We  solicited  comments  re¬ 
garding  grants'  objectives,  research  performance  and  results  from 
the  individual  researchers,  agency  program  officials,  and  in  some 
cases  the  renewal  grants'  peer  reviewers.  As  such,  much  of  the 
evaluative  information  in  this  report  is  based  on  the  responses 
provided  by  those  ostensibly  most  knowledgeable  about  the  grants. 
Our  independent  analysis  of  grant  information  was  done  by  compar¬ 
ing  and/or  contrasting  information  in  the  official  grant  files 
with  information  provided  by  the  researchers  and  others.  The 
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resulting  data,  while  representing  our  own  evaluation  of  the 
process,  does  not  include  evaluations  of  the  scientific  aspects 
of  the  grants  or  the  grants'  results. 

Our  study  approach  was  based  on  testing  certain  key  premises 
underlying  the  scientific  accountability  system.  In  this  system, 
assessing  a  researcher's  progress  and  results  on  a  preceding  grant 
by  peer  reviewers,  and  at  NSF  by  the  program  officer,  is  crucial 
for  determining  whether  additional  support  should  be  provided. 

One  premise  is  that  peer  reviewers  are  able  to  determine  the  qual¬ 
ity  and  significance  of  research  results  on  a  given  individual 
grant  and  thus  ensure  accountability  regarding  the  scientific 
merit  of  researchers'  work  under  that  grant.  At  NSF,  another 
premise  is  that  program  officers  are  sufficiently  knowledgeable  in 
the  areas  represented  in  the  proposals  they  review  to  assure  sci¬ 
entific  performance  accountability  by  evaluating  research  propo¬ 
sals  along  with  the  peer  review  comments.  The  study  approach  was 
also  designed  to  determine  whether  appreciable  differences  existed 
in  the  scientific  performance  accountability  provided  by  the  NSF 
system  as  compared  to  the  NIH  system,  and  within  NSF,  between  the 
ad  hoc  and  panel  systems.  Financial  accountability  was  not 
taken  into  consideration. 

OTHER  GAO  REPORTS 
AND  RELATED  STUDIES 

Within  the  last  5  years,  several  studies  and  reports  have 
been  issued  concerning  peer  review  and  grant  monitoring  at  NSF  and 
NIH.  Two  of  these  are  GAO  reports.  1/  One  contained  specific 
recommendations  to  NIH  to  establish  guidelines  concerning  the  con¬ 
tents,  the  review,  and  the  documentation  of  the  review  of  annual 
progress  reports  submitted  for  research  grants;  the  other  recom¬ 
mended  that  the  NSF  better  document  in  proposal  files  the  selec¬ 
tion  of  peer  reviewers  and  how  reviewers'  comments  are  handled, 
and  improve  internal  controls  to  assure  that  researchers  receive 
all  peer  review  comments  when  requested.  Neither  NSF  or  NIH  has 
fully  implemented  the  reports'  recommendations.  NSF  and  NIH  each 
have  conducted  studies  of  their  peer  review  systems.  A  study  of 
NSF' s  peer  review  system,  funded  by  NSF  and  conducted  under  the 
auspices  of  the  National  Academy  of  Sciences,  did  not  answer 
definitively  the  main  question  the  study  was  supposed  to  address: 
whether  NSF's  peer  review  system  is  an  "equitable"  one.  2/ 


^/"Better  Controls  Needed  Over  Biomedical  Research  Supported  by 
the  National  Institutes  of  Health,"  U.S.  General  Accounting 
Office,  HRD-76-58,  July  22,  1976;  and  "Accountability  in  the 
National  Science  Foundation’s  Review  Process  for  Grant  Awards 
Needs  Strengthening,"  U.S.  General  Accounting  Office, 
HRD-78-121,  November  17,  1978. 

2/"?eer  Review  in  the  National  Science  Foundation,  Phase  I  of  a 
Study,”  National  Academy  of  Sciences,  November  7,  1978. 
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NIH  conducted  a  lengthy  internal  review  of  its  peer  review 
system,  and  reached  some  of  the  same  conclusions  contained  in 
this  report  about  the  adequacy  of  the  peer  review  system.  1/ 
However,  the  methodology  used  to  conduct  NIH's  review  was  based 
primarily  on  anecdotal  information.  Also,  during  1980,  the 
National  Commission  on  Research  issued  two  reports  concerning 
scientific  accountability,  one  of  which  specifically  dealt  with 
peer  review  at  NSF  and  NIH.  2/  See  appendix  I  for  an  expanded 
discussion  of  related  reports. 


l/"Grants  Peer  Reviews  Report  to  the  Director,  NIH  Phase  I, 

December  1976,"  National  Institutes  of  Health;  and  "Grants  Peer 
Reviews  Report  to  the  Director,  NIH  Phase  II,  December  1978," 
National  Institutes  of  Health. 

2/"Review  Processes s  Assessing  the  Quality  of  Research  Propo¬ 
sals,"  The  National  Commission  on  Research,  May  1980. 
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CHAPTER  2 


RESEARCH  GRANT  SCIENTIFIC  PERFORMANCE 
ACCOUNTABILITY  SYSTEMS  OF  NSF 
AND  NIH  CAN  BE  IMPROVED 

Our  analysis  of  the  75  sample  grants  disclosed  that  NIH,  and 
NSF  to  a  lesser  extent,  rely  on  the  peer  review  process  to  provide 
scientific  performance  accountability  of  basic  research  grants. 

NIH  peer  reviewers  evaluate  (1)  the  scientific  merit  of  the  pro¬ 
posed  research,  (2)  the  past  productivity  of  the  researcher,  and 
(3)  the  researcher's  performance  on  the  immediately  preceding 
grant,  for  renewal  proposals.  At  NSF,  however,  peer  reviewers 
perform  their  function  somewhat  differently.  In  accordance  with 
NSF  guidelines,  peer  reviewers  only  evaluate  the  scientific  merit 
of  the  research  and  the  past  productivity  of  the  researcher — they 
generally  do  not  evaluate  performance  on  the  immediately  preceding 
grant.  Additionally,  even  if  evaluation  of  the  immediately  pre¬ 
ceding  grant  were  required,  NSF's  proposal  instructions  do  not 
facilitate  such  an  evaluation.  More  specific  guidelines  to  peer 
reviewers  and  for  renewal  proposal  requirements  are  needed  at  NSF 
because  in  many  cases  different  program  officers  and  peer  re¬ 
viewers  evaluate  the  renewal  proposal  than  evaluate  the  preceding 
grant  proposal. 

NIH  uses  peer  reviewer  evaluations  more  extensively  than  NSF 
does.  NIH  evaluations  have  a  greater  effect  on  grant  awards  than 
NSF's  because  of  the  greater  flexibility  NSF  program  officials 
have  in  making  award  decisions.  Additionally,  NSF's  practice  of 
only  releasing  peer  reviewer  evaluations  to  researchers  upon  re¬ 
quest  does  not  maximize  the  use  of  these  evaluations.  Our  analy¬ 
sis  of  the  7:5  grants  shows  that  these  evaluations  could  be  useful 
to  researchers  if  they  were  routinely  released.  Further,  for 
many  NSF  grants  which  were  panel  reviewed,  no  summary  of  panel 
discussion  was  available  to  forward  to  the  researcher. 

NSF  and  NIH  do  not  uniformly  monitor  or  evaluate  research 
grants.  Program  officials  vary  in  their  approach  to  both  these 
functions  and  they  sometimes  rely  on  peer  reviewers  to  perform 
them.  Additionally,  the  universities  we  visited  do  not  monitor 
grant  progress  or  evaluate  grant  results.  They  only  review  grant 
proposals  for  administrative  matters  and  adherence  to  university 
policy. 

SCIENTIFIC  PERFORMANCE  ACCOUNTABILITY 


Science  must  be  fundamentally  accountable  to  society  in  that 
it  uses  society's  resources  to  seek  continually  greater  under¬ 
standing  of  nature.  When  that  happens,  science  is  using  resources 
as  intended  and  is  being  accountable  to  its  patrons.  If  that 
does  not  happen,  then  society  is  not  getting  what  it  should  from 
science  for  the  resources  invested. 
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NSF  and  NIH  attempt  to  maintain  accountability  for  the 
science  they  support  in  basic  research  grants  through  a  set  of 
practices  and  procedures,  which  taken  together  form  a  process  for 
assuring  scientific  performance  that  we  call  the  scientific  per¬ 
formance  accountability  process.  We  are  interested  primarily  in 
evaluating  the  systems  and  procedures  used  by  the  two  agencies 
to  assure  that  their  basic  research  grants  foster  good  science. 

However,  the  accountability  process  should  not  be  viewed 
(and  we  do  not  view  it)  as  an  end  in  itself.  The  process  is 
merely  a  means  to  an  end — with  the  end  being  supporting  the  best 
science  possible  in  the  basic  research  grants  NSF  and  NIH  award. 
The  process  should  serve  science;  it  should  aid  in  advancing 
science. 

No  single  regulation  or  set  of  instructions  defines  the 
scientific  performance  accountability  process  used  by  NSF  or  NIH. 
We  use  the  accountability  process  to  describe  and  organize  a  set 
of  customs,  practices,  and  procedures  these  agencies  use.  The 
process  consists  of  five  major  elements: 

— proposal  submission; 

— peer  review  process; 

— award  decision; 

— monitoring  the  research;  and 

— evaluating  the  research. 

Peer  review  of  proposals  for  basic  research  is  used  by  NSF 
and  NIH  to  solicit  expert  advice  from  the  scientific  community  to 
help  determine  which  research  proposals  merit  funding.  Agency 
program  officers  are  responsible  for  the  actual  decisions  regard¬ 
ing  which  proposals  are  funded  and  for  monitoring  ongoing  re¬ 
search  to  insure  that  work  is  carried  out  under  the  terms  of  the 
grant.  Acceptance  for  publication  in  the  open  scientific  litera¬ 
ture  provides  for  dissemination  of  scientific  results  and  some 
measure  of  evaluation,  e.g.,  articles  in  leading  journals  are 
subjected  to  peer  review  prior  to  acceptance  for  publication. 

The  grant  award  process  at  both  NSF  and  NIH  depends  heavily 
on  peer  review  to  provide  scientific  accountability  and  on  agency 
program  officers  to  make  the  basic  award  decisions  on  proposals 
for  research  grants.  However,  the  operation  of  the  grant  award 
process  and  peer  review  at  the  two  agencies  differs  significantly, 
the  most  significant  being  the  authority  of  the  peer  reviewers 
at  NIH  as  compared  to  the  authority  of  the  program  officers  at 
NSF.  Table  2  compares  the  grant  award  systems  of  NSF  and  NIH. 
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Table  2 


Comparing  the  NSF  and  NIH  Systems 
Used  to  Review  Research  Proposals 


NSF 


Process 

Proposal  requirements 

Ad  hoc 

Panel 

NIH 

All  proposals  must  show 

long-range  objectives 
specific  aims  (objectives)  to 
be  accomplished  during 

yes 

yes 

yes 

proposal  grant  period 

Renewal  proposals  must  show 
objectives  of  immediately 

no 

no 

yes 

preceding  grant 
summary  of  progress  on 

no 

no 

yes 

immediately  preceding  grant 
specific  publications  that 
resulted  from  immediately 

yes 

yes 

yes 

preceding  grant 

Peer  reviewers 

no 

no 

yes 

Make  the  scientific  merit  decisions 

on  proposals 

no 

no 

yes 

Determine  budget  amounts 

Required  to  comment  on  immediately 

no 

no 

yes 

preceding  grant 

no 

no 

yes 

Prioritize  proposals 

Comments  automatically  released 

no 

yes  a/ 

yes 

to  researcher 

Award  decision 

Can  be  made  by  program  officer  even 
if  peer  reviewers  do  not  approve 

no 

no 

yes 

the  scientific  merit  of  a  proposal 

yes 

yes 

no 

Monitoring  the  research 

Progress  reports  required 

yes 

yes 

yes 

Site  visits  made 

Evaluating  the  research 

yes 

yes 

yes 

Final  project  report 

required  on  every  grant 

Final  project  report 

required  upon  completion 

yes 

yes 

no 

of  a  line  of  research 

no 

no 

yes 

Submission  of  publications  required 

yes 

yes 

yes 

a/Some  NSF  panels  prioritize  proposals. 
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PROPOSAL  SUBMISSION 


The  research  proposal  presents  the  researcher's  past  per¬ 
formance  and  describes  a  program  of  planned  research  which  is 
tailored  to  determine  whether  he  or  she  should  receive  a  research 
grant  from  NSF  or  NIH.  Proposals  are  submitted  for  either  a  new 
research  project  or  to  request  continued  funding  for  an  ongoing 
project.  The  latter  is  called  a  renewal. 

After  the  researcher  has  prepared  the  proposal,  it  is  submit 
ted  to  a  number  of  university  officials  for  review  and  signoff. 
Proposals  are  generally  reviewed  by  the  researcher's  department 
chairman  and/or  university  dean  responsible  for  the  department, 
and  by  an  office  responsible  for  the  administrative  aspects  of 
the  university's  externally  sponsored  research.  Universities  are 
not  required  by  either  NSF  or  NIH  proposal  preparation  instruc¬ 
tions  to  review  proposals  for  scientific  merit,  or  for  the  need 
or  relevance  of  the  proposed  research. 

Both  NSF  and  NIH  require  researchers  to  submit  proposals 
describing  the  research  to  be  supported  by  the  grant  being  sought 
With  minor  exceptions,  the  proposals  for  the  grants  in  our  sample 
included  all  the  information  required  by  NSF  or  NIH.  The  univer¬ 
sities  we  visited  review  proposals  for  administrative  and  policy 
matters,  but  not  for  technical  adequacy  or  the  need  for  the 
research . 

THE  PEER  REVIEW  PROCESS 


At  both  NSF  and  NIH,  review  of  researchers'  proposals  by 
outside  scientists  (called  peer  review)  is  considered  the  most 
important  and  most  effective  means  to  assure  scientific  perform¬ 
ance  accountability.  Almost  all  proposals  for  basic  research 
submitted  to  the  two  agencies  are  subjected  to  some  form  of  peer 
review. 

"The  words  'peer  review'  [mean]  review  by  scientists 
who  are  actively  engaged  in  research,  who  are  not 
employed  by  a  funding  agency,  and  who  have  the  research 
experience  and  achievement  which  will  permit  them  to 
make  discerning  judgments  on  the  scientific  merits  of 
[research]  proposals."  1/ 

Peer  review  is  supposed  to  provide  advisory  information  on 
the  scientific  merit  or  quality  of  the  research  being  proposed, 
the  track  record  or  past  productivity  of  the  researcher,  and  the 
reasonableness  of  the  proposed  budget. 


l_/National  Commission  on  Research,  "Review  Processes:  Assessing 
the  Quality  of  Research  Proposals,"  May  1980,  p.  3. 
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NSF  guidelines  do  not  distinguish  proposals  for  renewal 
grants  from  new  project  grants  and  peers  are  simply  asked  to  eval¬ 
uate  a  researcher's  recent  accomplishments — they  are  not  specifi¬ 
cally  asked  to  comment  on  the  progress  of  the  immediately  preced¬ 
ing  grant.  Our  analysis  of  NSF  peer  review  comments  shows  that, 
while  peer  reviewers  are  evaluating  the  researcher's  overall  track 
record,  they  generally  do  not  comment  on  the  budget  resources 
and  environment  or  the  progress  under  the  immediately  preceding 
grant. 

NIH  guidelines  make  an  additional  distinction  which  NSF ’ s 
do  not.  For  renewal  grant  proposals,  NIH  peer  reviewers  are 
specifically  asked  to  comment  on  the  progress  of  the  immediately 
preceding  grant.  Our  analysis  of  NIH  peer  review  comments  indi¬ 
cates  that  peer  reviewers  are  evaluating  all  of  the  elements  of 
the  proposal  they  are  requested  to. 

Evaluating  scientific  merit 

Scientific  merit  is  one  element  that  NSF  and  NIH  peer  review¬ 
ers  are  requested  to  evaluate  in  proposals.  They  do  this  by  exam¬ 
ining  the  researcher's  goals  and  the  research  work  plan  outlined 
in  the  research  proposal.  Research  goals,  known  as  objectives  or 
specific  aims,  are  described  in  the  pr  posal  in  terms  of  their 
significance  and  relationship  to  current  knowledge.  Work  plans 
include  experiments,  methods,  and  procedures  intended  to  accomp¬ 
lish  these  goals.  Peer  reviewers  evaluate  the  scientific  merit 
of  goals  and  work  plans  by  considering  their  strengths,  weak¬ 
nesses,  originality,  creativity,  adequacy  of  experimental  design, 
etc.  Our  review  shows  that  peer  reviewers  evaluated  the  scienti¬ 
fic  merit  of  all  the  proposals  included  in  our  sample. 

Evaluating  the  researcher  *s  productivity 

Each  researcher's  productivity  or  track  record  is  another 
element  which  NSF  and  NIH  peer  reviewers  are  requested  to  evalu¬ 
ate.  Research  proposals  include  biographical  sketches  of  re¬ 
searchers,  information  on  their  qualifications  to  perform  the  pro¬ 
posed  research  and  lists  of  their  scientific  publications.  NSF 
requires  a  list  of  the  researcher's  publications  for  the  preceding 
5  years.  NIH  requires  a  list  of  all  publications,  or  if  this  is 
not  possible,  a  list  of  their  most  representative  publications. 
Peer  reviewers  evaluate  this  information  in  terms  of  the  compe¬ 
tence  of  the  researcher  to  complete  the  proposed  research  and 
the  likelihood  of  their  accomplishing  the  proposed  research. 

Our  review  shows  that  with  two  exceptions  peer  reviewers  evaluated 
the  track  record  of  all  the  researchers  included  in  our  sample. 

Peer  review  of  researchers'  track  records  involves  retro¬ 
spective  analysis  of  the  productivity  of  their  research.  Peer 
reviewers  do  not  review  researchers'  productivity  in  terms  of 
their  ability  to  obtain  prior  grants,  but  rather  in  terms  of 
publications.  As  an  NSF  official  testified  during  congressional 
hearings,  "...in  basic  research  one  of  the  strongest  tools  for 
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evaluation  is  publication  in  the  open  literature  and  critical  re¬ 
view  by  other  scientists."  Publication,  however,  is  not  the  only 
factor  determining  a  researcher's  track  record.  Other  factors  in¬ 
clude  dissemination  of  research  results  through  such  means  as 
theses,  dissertations,  speeches,  and  informal  communications  within 
the  research  community. 

Given  the  importance  of  track  record,  experienced  researchers 
would  appear  to  have  an  advantage  in  obtaining  grants.  NSF  and 
NIH  have  studied  this  problem.  A  1977  study,  funded  by  NSF,  was 
made  to  determine  the  impact  of  various  factors  such  as  profes¬ 
sional  age  on  NSF's  grant  process.  The  study  showed  that  profes¬ 
sional  age  has  almost  no  effect  on  peer  review  ratings  or  funding 
decisions,  and  that  no  systematic  discrimination  occurs  at  NSF 
against  noneminent  scientists.  NSF  award  statistics  for  fiscal 
year  1977  support  these  conclusions.  Of  the  11,158  researchers 
funded,  3,557  (32  percent)  were  funded  for  the  first  time. 

A  1978  study  of  the  NIH  peer  review  system  was  made  using 
data  gathered  through  public  hearings,  letters  soliciting  com¬ 
ments,  and  a  survey  of  all  1975-76  NIH  grant  review  groups.  This 
study  generally  concluded  that  the  NIH  peer  review  system  does 
not  discriminate  against  inexperienced  researchers. 

These  conclusions  were  echoed  by  25  NSF  program  officers 
interviewed  in  1978  by  congressional  staff  members.  1/  Two-thirds 
felt  that  both  peer  reviewers  and  program  officers  attempt  to  give 
young  investigators  a  "break."  Specifically,  they  suggested  that 
if  an  inexperienced  researcher's  proposal  is  of  borderline  quality, 
it  will  be  funded  at  a  modest  level. 

Of  the  75  grants  we  reviewed  in  our  sample  (25  NIH  and  50 
NSF),  10  were  first  time  awards  to  beginning  researchers;  3  (12 
percent)  were  awarded  by  NIH  and  7  (14  percent)  were  awarded  by 
NSF. 

Evaluating  scientific  merit  vs.  track  record 

No  clear  pattern  of  emphasis  regarding  scientific  merit  ver¬ 
sus  track  record  emerged  from  our  review.  NSF  and  NIH  peer  review 
instructions  make  no  distinctions  on  the  relative  importance  of 
either  factor.  NSF  funding  guidelines,  however,  state  that  all 
funding  considerations  are  "predicated  upon  the  assumption  that... 
competent  performance  will  be  the  minimum  expectation."  Since 
track  record  indicates  performance  potential,  this  guideline 


1/Interview  of  National  Science  Foundation  Program  Of f icers-Final 
Report-prepared  by  the  Subcommittee  on  Science,  Research  and 
Technology  of  the  Committee  on  Science  and  Technology,  U.S. 
House  of  Representatives,  Ninety-sixth  Congress,  first  session, 
July  1979. 
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suggests  that  track  record  may  be  viewed  as  more  important  than 
scientific  merit. 

Researchers  and  agency  officials  expressed  differing  views 
on  the  question  of  whether  scientific  merit  or  track  record  was 
emphasized  more  in  proposal  evaluations.  When  asked  which  they 
weighed  more  when  peer  reviewing  other  scientists'  proposals, 
the  majority  of  the  75  researchers  involved  in  our  sample  grants 
who  responded,  as  table  3  indicates,  placed  more  emphasis  on  the 
scientific  merit  of  the  proposal. 

Table  3 


Factors  Wei< 

ihed  in  Peer  Reviewing 

Other  Sc: 

Lentists'  Proposals 

NSF- 

NSF- 

Factor 

NIH 

Panel  Ad  Hoc 
-(Percentages) - 

Overall 

The  researcher's  reputation 

(track  record) 

The  proposed  research 

12 

9 

24 

16 

(scientific  merit) 

34.5 

32 

55 

42 

Both  are  considered  equal 

No  peer  review  performed 

19 

41 

7 

21 

for  NSF  and  NIH 

34.5 

106.0 

18 

M 

14 

21 

IoS 

However,  most  agency  officials  said  that  they  had  no  set  weights 
for  track  record  versus  scientific  merit  and  that  the  emphasis 
changes  for  each  individual  case.  These  responses  indicate  that 
researchers  and  agency  officials  use  their  individual  judgment 
in  weighing  scientific  merit  against  track  record  when  evaluating 
proposals. 

Funding  received  despite  inadequate  proposals 

Peer  reviewers  for  the  grants  we  reviewed  also  placed  vary¬ 
ing  amounts  of  emphasis  on  scientific  merit  and  track  records. 
Scientific  merit  was  stressed  on  first-time  awards  and  track  rec¬ 
ord  was  emphasized  for  some  experienced  researchers.  For  example, 
although  one  proposal  (NSF  ad  hoc)  submitted  by  an  experienced 
researcher  contained  no  work  plan--an  NSF  requirement — peer  re¬ 
viewers  did  not  comment  on  the  omission  and  recommended  that  the 
proposal  be  funded.  The  program  official  responsible  for  this 
grant  stated  that  on  the  basis  of  his  past  experience,  he  was 
certain  the  researcher  would  produce  something  worthwhile  even 
without  a  work  plan. 

Another  proposal  (NSF  panel)  was  criticized  by  peer  review¬ 
ers  for  its  vagueness  and  its  lack  of  a  work  plan.  Cfrie  peer 
reviewer  summarized  his  evaluation  of  the  proposed  work  as  "ill- 
considered;  this  halfhearted  approach  to  such  a  range  of  difficult 
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and  challenging  problems  cannot  be  taken  seriously."  However, 
another  peer  reviewer's  comments  suggest  the  eminence  of  the  re¬ 
searcher  compensated  for  his  weak  proposal.  As  this  peer  put 
it. 


"In  conclusion  it  appears  that  little  effort  was  made 
in  writing  this  proposal  to  even  hint  at  the  possible 
approaches  he  intends  to  take  on  these  very  important 
questions .  This  is  not  to  say  that  the  work  would  not 
be  worth  supporting  on  the  basis  of  past  experience 
with  the  investigator." 

When  queried  about  the  lack  of  a  work  plan,  the  researcher  ex¬ 
plained  it  was  implicit  in  the  proposed  research  which  involved 
equation  solving  and  theorizing.  The  program  official  respon¬ 
sible  for  this  grant  agreed.  He  added  that  he  and  the  peer  re¬ 
viewers  knew  the  kind  of  research  performed  by  this  researcher, 
and  that  he  would  produce  publications. 

In  a  third  instance,  a  peer  reviewer  characterized  the  pro¬ 
posal  ( NSF  panel)  as  vague  on  both  its  future  objectives  and  past 
accomplishments.  However,  he  concluded  that  since  he  was  not 
sure  what  the  researcher  proposed  to  do,  all  he  could  do  was  look 
at  the  researcher's  name  and  publication  record  and  recommend 
that  the  proposal  be  funded. 

The  importance  of  a  researcher's  track  record  was  very 
apparent  in  another  instance  (NSF  panel).  In  this  instance,  the 
name  of  an  eminent  scientist  may  have  been  used  to  generate  fund¬ 
ing  for  a  weak  proposal.  One  peer  reviewer  mentioned  his  surprise 
that  the  experienced  scientist's  name  was  associated  with  such  a 
weak  proposal.  The  experienced  scientist  told  us  that  he  was 
unaware  that  he  was  listed  as  a  co-researcher.  The  researcher 
explained  that  the  experienced  scientist  only  helped  prepare  the 
proposal  and  had  not  worked  on  the  research.  The  program  officer 
responsible  for  this  grant  stated  he  was  unaware  of  the  experi¬ 
enced  scientist  being  a  co-researcher  in  name  only.  The  above 
examples  show  that  some  researchers  because  of  their  perceived 
scientific  eminence  can  get  NSF  funds  with  weak  or  vague  propo¬ 
sals,  or  without  complying  with  the  proposal  requirements. 

At  NIH,  we  found  no  examples  of  proposals  in  our  sample  get¬ 
ting  funded  which  did  not  comply  with  the  proposal  requirements 
or  which  were  weak  or  vague. 

Evaluating  preceding  grant  performance 

Progress  on  a  researcher's  immediately  preceding  grant  is  a 
critical  factor  in  awarding  grants  to  continue  long-term  research 
projects.  Its  importance  was  summarized  in  March  1978  congres¬ 
sional  testimony  by  a  top  NSF  official  as  follows: 
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"In  the  basic  research  areas,  most  investigators 
submit  renewal  proposals  which  are  required  to 
contain  a  summary  of  progress  under  the  preceding 
award;  this  is  evaluated  by  peer  reviewers  and 
program  staff  as  a  crucial  part  of  the  review 
process . 

Our  peer  review  system  relies  heavily  on  evidence 
in  the  immediate  past  of  research  productivity  of 
significance.  This  does  represent  an  ongoing 
monitoring  of  whether  the  recently  granted  Federal 
funds  were  well  used  and  that  is  a  heavy  consti¬ 
tuent  in  deciding  whether  additional  funds  should 
be  directed  that  way. 

We  were  contending  that  the  peer  process  includes 
an  implicit  retroactive  analysis  of  productivity 
on  earlier  grants  and  that  we  consider  the  analy¬ 
sis  to  be  an  important  criterion  of  whether  money 
is  being  well  used." 

NSF 

While  the  above  quotation  indicates  the  importance  of  cri¬ 
tiquing  the  researcher's  performance  on  the  immediately  preceding 
grant,  NSF  still  does  not  specifically  request  peer  reviewers  to 
do  so,  although  the  scientific  merit  of  the  proposal  and  the 
researcher's  general  track  record  are  evaluated.  More  specific 
guidance  to  peer  reviewers  leading  to  more  specific  peer  review 
comments  is,  in  our  opinion,  extremely  important  because  (1)  pro¬ 
gram  officers  may  not  have  the  expertise  to  make  such  evaluations 
and  (2)  in  many  instances,  different  program  officers  and  peer 
reviewers  are  evaluating  the  renewal  proposal  than  evaluated  the 
preceding  proposal. 

Our  analysis  of  the  peer  review  comments  for  the  27  NSF 
grants  which  were  renewed  (of  the  50  reviewed)  showed  that  evalu¬ 
ative  comments  concerning  the  grant  preceding  the  renewal  were 
made  in  only  22  percent  of  the  cases.  In  only  6  of  the  27  cases 
was  performance  on  the  immediately  preceding  grant  evaluated  as 
a  factor  in  awarding  the  renewal. 

A  renewal  proposal  usually  contains  a  summary  of  progress 
made  under  the  preceding  grant  which  is  presumably  read  by  the 
peer  reviewers  as  part  of  proposal  review.  However,  if  the  peer 
review  comments  furnished  to  the  NSF  program  officers,  who  make 
the  award  decision,  are  silent  regarding  the  progress  of  the 
immediately  preceding  grant,  these  officers  either  have  to  guess 
at  what  the  peer  reviewers  thought  about  the  preceding  grant  or 
make  their  judgment  as  to  the  progress.  Both  options  involve 
risk.  Also,  in  5  of  the  21  cases  where  evaluation  of  performance 
on  the  immediately  preceding  grant  by  the  peer  reviewers  was  not 
evident,  the  cognizant  NSF  program  officers  stated  that  they 
would  not  have  had  the  expertise  necessary  to  serve  as  a  peer 
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reviewer  for  the  renewal  proposal  in  question.  As  a  result, 
there  is  no  evidence  that  evaluation  of  the  researcher's  perform¬ 
ance  under  the  immediately  preceding  grant  occurred  in  these 
five  cases. 

NIH 

At  NIH,  in  addition  to  scientific  merit  and  general  track 
record,  peer  reviewers  also  evaluate  a  reseacher's  performance  on 
the  immediately  preceding  grant  when  a  research  project  is  funded 
by  a  series  of  grants.  Of  the  25  NIH  grants  reviewed,  19  were  re¬ 
newed  to  continue  the  same  line  of  research.  In  18  of  these  19 
renewals,  analysis  of  the  peer  review  comments  indicated  that  the 
researcher's  performance  on  the  immediately  preceding  grant  was 
evaluated  in  critiquing  the  renewal  proposal.  Hence,  the  peer 
review  process  has  a  cyclical  nature  for  long-term  research 
projects  in  that  research  goals  and  preceding  grant  performance 
are  periodically  evaluated  through  renewal  proposals  as  a  basis 
for  continued  support. 

N5F  proposal  requirements  do  not  assure 

adequate  information  for  easy  evaluation  of 

preceding  grant  performance 

NSF  proposal  requirements  do  not  reflect  the  importance  of 
evaluating  the  performance  of  the  immediately  preceding  grant. 

NSF  instructions  to  researchers  leave  much  to  the  researcher's 
judgment  as  to  progress  report  content  and  researchers  are  not 
required  to  identify  any  publications  which  resulted  from  the 
immediately  preceding  grant.  Proposals  which  easily  identify 
preceding  grant  results  (see  definition  on  p.  8)  are  particu¬ 
larly  important  because  (1)  they  often  are  the  only  source  of 
these  results  at  the  time  of  renewal  and  (2)  different  peer 
reviewers  usually  review  successive  grants. 

NSF  only  requires  renewal  proposals  to  include  a  "summary  of 
progress  to  date  (on  the  preceding  grant)  and  its  relation  to  the 
proposed  work."  Researchers,  when  preparing  renewal  proposals, 
are  further  instructed  to  assume  peer  reviewers  will  not  have 
access  to  previous  proposals. 

In  contrast,  NIH  currently  requires  specific  information  on 
the  preceding  grant  to  include  the  time  covered  since  the  last 
peer  review;  the  preceding  grant's  specific  aims  (research  objec¬ 
tives);  a  "succinct  account  of  published  and  unpublished  results 
indicating  progress  toward  their  (research  objectives)  achieve¬ 
ment";  a  discussion  of  the  importance  of  results;  a  discussion 
of  any  changes  in  research  objectives  since  the  last  peer  review; 
and  a  list  of  titles  plus  complete  references  to  all  publications, 
completed  manuscripts,  patents,  invention  reports,  and  other 
printed  materials  that  resulted  since  the  last  peer  review. 
Additionally,  copies  of  publications  and  completed  manuscripts 
are  to  be  appended  to  the  renewal  proposal. 
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Our  analysis  of  50  NIH  and  NSF  renewal  proposals  showed  that 
results  of  the  preceding  grant  were  not  included  in  1  of  the  23 
submitted  to  NIH  and  in  3  of  the  27  to  NSF.  Furthermore,  of  the 
remaining  proposals,  all  of  NIH's  contained  a  progress  report 
section  that  was  clearly  labeled  and  identifiable.  However,  in 
at  least  seven  of  the  NSF  proposals,  researchers  interspersed 
results  from  their  preceding  grants  throughout  the  general  intro¬ 
ductory  and  other  material  in  the  proposals  instead  of  present¬ 
ing  the  results  separately. 

According  to  several  program  officials  and  a  peer  reviewer, 
the  lack  of  specific  NSF  reporting  requirements  for  results  on 
the  immediately  preceding  grant  makes  performance  evaluation 
difficult.  NSF's  failure  to  require  reporting  of  research  objec¬ 
tives  for  the  preceding  grant,  for  example,  can  make  difficult 
determining  how  the  proposed  work  relates  to  work  under  the  pre¬ 
vious  grant.  This  problem  was  succinctly  summarized  by  one  peer 
reviewer  of  an  NSF  renewal  proposal  who  wrote: 

"Furthermore,  the  other  proposal  (sample  grant) 
also  dealt  with  pipeline  processing.  I  have  no 
way  of  comparing  the  current  proposal  (renewal) 
with  the  other  one  and  find  out  what  is  new." 

Another  peer  reviewer  for  a  different  NSF  proposal  had  problems 
with  this  format  and  made  the  following  critical  comments  about 
the  renewal  proposal: 

"Specific  objectives. . .are  not  described  in  as  great 
detail  by  the  proposer  as  this  reviewer  would 
wish."  [As  this  is  a  renewal  request,  some  of  the 
reviewer's  questions  may  have  been  addressed  in  an 
earlier  proposal  or  progress  report..  ] 

Yet  there  is  no  clear  statement  of  accomplish¬ 
ments  during  the  prior  grant  on  which  to  base 
a  judgment  on  how  much  of  what  is  talked  about 
here  is  likely  to  be  finished  during  the  re¬ 
newal  period." 

To  determine  if  research  objectives  were  the  same  for  succes¬ 
sive  grants,  we  compared  sample  grant  objectives  (specific  aims 
for  NIH)  with  renewal  grant  objectives.  In  only  2  of  the  23 
renewal  proposals  submitted  to  NIH  did  the  research  objectives 
appear  to  be  the  same.  This  sameness  occurred  in  7  of  the  27 
renewal  proposals  submitted  to  NSF.  The  work  plans  for  these 
grants  did  change.  (See  chapter  3  for  a  discussion  of  research 
objectives.)  However,  if  peer  reviewers  do  not  have  information 
to  compare  research  objectives  under  successive  grants,  deter¬ 
mining  how  proposed  work  differs  from  the  work  under  the  preceding 
grant  is  difficult.  Current  NIH  proposal  review  procedures  pro¬ 
vide  information  on  research  objectives  for  successive  grants. 

In  contrast,  NSF  procedures  do  not  insure  peer  reviewers  will 
have  information  on  research  objectives  for  successive  grants. 
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NSF’s  not  requiring  researchers  to  identify  the  publications 
resulting  from  their  preceding  grants  also  creates  performance 
evaluation  problems.  NSF  currently  only  requires  researchers  to 
list  all  thwir  publications  for  the  preceding  5  years.  While  this 
does  indicate  general  productivity,  relating  publications  to  a 
specific  grant  from  a  general  listing  is  difficult.  This  diffi¬ 
culty  is  increased  if  a  researcher  has  several  concurrent  grants. 
Fifty-two  of  the  75  researchers  included  in  our  review  had  other 
grants  concurrent  with  their  sample  grant.  Eighteen  of  these  52 
had  concurrent  NIH  and  NSF  grants. 

Our  analysis  of  renewal  proposals  showed  that  many  did  not 
identify  publications  resulting  from  the  preceding  grant.  This 
was  particularly  true  with  NSF  proposals,  as  our  analysis  showss 

Table  4 


Publications  Resulting  from  the  Preceding  Grant 


NIH 

NSF 

Total 

Identified 

19 

10 

29 

Not  identified 

4 

15 

19 

ffo  renewal  sought 

Renewal  proposal  not 

2 

23 

25 

analyzed 

0 

2 

2 

1 

Total 

25 

50 

75 

One  grant  (NSF  ad  hoc)  exemplifies  the  problem  caused  by  not 
identifying  the  preceding  grant's  publications.  In  this  example, 
the  immediately  preceding  grant  did  not  produce  any  publications. 
However,  the  renewal  proposal  contained  a  listing  of  publications 
from  the  researcher's  concurrent  grants,  but  the  proposal  did  not 
identify  the  grants  from  which  the  publications  resulted.  Offi¬ 
cials  did  not  recognize  that  the  grant  the  researcher  wanted  to 
renew  did  not  produce  any  of  the  publications  identified  in  the 
renewal  proposal,  and  awarded  the  researcher  another  grant  to 
continue  the  research  project.  The  cognizant  program  officer 
said  that  this  would  not  have  occurred  if  he  knew  there  were  no 
publications.  A  peer  reviewer  echoed  this  observation  by  stating 
that  unless  a  researcher  voluntarily  identifies  publications 
resulting  from  the  preceding  grant,  reviewers  have  no  means  to 
assess  performance. 

In  total,  55  percent  (32  of  58)  of  the  peer  reviewers  we 
contacted  stated  that  having  the  results  of  the  immediately  pre¬ 
ceding  grant  as  related  to  the  grant's  objectives  would  be  useful 
information. 

Publications  are  not  always  available  when  researchers  submit 
their  renewal  proposals.  Renewal  proposals  must  be  submitted  6 
to  9  months  prior  to  the  expiration  of  the  preceding  grant.  The 
progress  report  in  a  renewal  proposal  clearly  showing  the  results 
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of  the  preceding  grant  often  provides  the  best  source  of  preceding 
grant  results  at  the  time  of  renewal. 

We  compared  the  dates  that  renewal  proposals  were  submitted 
to  NIH  and  NSF  with  the  dates  of  the  publications  from  the  pre¬ 
vious  grant  to  determine  if  the  publications  were  available  to 
the  peer  reviewers  evaluating  the  renewal  proposal.  At  NIH,  in 
13  of  the  19  cases  (68  percent)  where  the  renewal  was  awarded, 
publications  from  the  previous  grant  were  available  when  the  peer 
reviewers  evaluated  the  renewal  proposal.  At  NSF,  in  only  10  of 
the  27  cases  (37  percent)  where  the  renewal  was  awarded  were  pub¬ 
lications  available.  Officials  at  both  agencies  stated  this 
timing  problem  is  ameliorated  by  listing  pending  publications  or 
appending  manuscripts,  preprints  or  reprints  of  articles  from  the 
preceding  grant  to  renewal  proposals.  We  found  that  such  infor¬ 
mation  was  included  in  5  of  the  6  NIH  cases  where  publications 
were  not  available,  but  was  included  in  only  9  of  the  17  NSF  cases 
where  publications  were  not  available.  Thus,  the  peer  reviewers 
did  not  have  printed  results  to  review  from  the  preceeding  grant 
for  one  NIH  renewal  proposal  and  for  eight,  or  30  percent,  of 
the  NSF  renewal  proposals  of  grants  in  our  sample. 

Additionally,  as  discussed  below,  since  different  peer  re¬ 
viewers  typically  evaluate  successive  NSF  grants  awarded  to 
pursue  a  long-term  line  of  research,  a  clear,  easily  identified 
presentation  of  preceding  grant  results  would  improve  proposal 
evaluation. 

Effect  of  NSF  program  officer 

and  peer  reviewer  turnover 

The  NSF  program  officers  and  peer  reviewers  who  evaluate  a  re¬ 
searcher's  renewal  proposal  frequently  are  different  from  those 
who  evaluated  the  proposal  from  the  immediately  preceding  grant. 
This  lack  of  continuity  in  evaluative  personnel  can  affect  the 
renewal  proposal  evaluation. 

Many  of  NSF's  program  officers  are  "rotators"  who  serve  in 
this  capacity  for  a  period  of  1  to  2  years.  As  a  result,  the 
program  officer  responsible  for  awarding  a  renewal  grant  may  not 
have  been  responsible  for  awarding  and  monitoring  the  preceding 
grant  and  might  not  be  cognizant  of  the  researcher's  performance 
on  the  preceding  grant.  This  lack  of  continuity  in  program  offi¬ 
cers  can  be  illustrated  by  two  NSF  ad  hoc  grant  examples.  In 
one  case,  five  different  program  officers  were  involved  over  a 
5-year  span  covering  three  NSF  ad  hoc  grant  awards  (the  last  two 
being  renewals  of  the  preceding  grant)  totaling  about  $80,000.  1/ 
These  three  grants  all  had  identical  objectives  and  as  of  the 
end  of  the  second  grant  (our  sample  grant)  no  peer  reviewed 


1/A  fourth  proposal--the  third  renewal — had  been  submitted  but 
not  awarded  at  the  time  of  our  review. 


25 


publications  had  been  produced.  The  researcher  had,  however, 
published  non-peer  reviewed  technical  papers.  These  papers  were 
not  included  with  the  renewal  proposal  and  were  not  furnished  to 
the  peer  reviewers  by  the  program  officer.  One  of  the  peer 
reviewers  for  the  third  grant  (the  second  renewal)  did  request 
these  papers  from  the  program  officer  before  making  his  evalua¬ 
tion.  The  program  officer  turnover  in  this  case  was  compounded 
with  peer  reviewer  turnover.  Of  the  20  peer  reviewers  used  to 
evaluate  these  three  grants,  none  reviewed  all  three. 

In  the  second  case,  five  different  program  officers  were 
involved  over  a  5-1/2  year  span  covering  two  NSF  ad  hoc  grant 
awards  totaling  $525,000.  Also,  as  in  the  first  example,  peer 
reviewer  continuity  was  totally  lacking  between  the  two  grants. 
None  of  the  11  different  peer  reviewers  evaluated  both  grants. 

In  total  for  the  50  NSF  grants  we  reviewed,  we  were  only 
able  to  talk  to  the  cognizant  program  officer  1/  in  28  instances 
(56  percent).  In  21  of  the  22  other  instances,  the  program 
officer  had  left  NSF;  in  one  instance  he  was  on  sabbatical.  At 
NIH,  we  were  able  to  talk  to  the  cognizant  program  officer  for 
20  of  the  25  grants  renewed  (80  percent). 

Lack  of  continuity  is  also  a  concern  in  NSF's  selection  of 
peer  reviewers.  For  the  27  grants  renewed,  we  examined  the  names 
of  the  peer  reviewers  for  the  grant  included  in  our  review  and 
its  renewal.  In  one  case,  three  of  the  same  peer  reviewers  were 
used,  in  two  cases  two  of  the  same  reviewers  were  used,  in  14 
cases  (52  percent)  only  one  of  the  reviewers  was  the  same  and  in 
10  cases  (37  percent)  totally  different  reviewers  were  used. 

At  NIH,  peer  reviewer  continuity  is  less  of  a  problem  because 
of  the  tenure  and  continuity  of  membership  of  the  peer  review 
group.  There  will  generally  be  individuals  in  the  peer  review 
group  to  review  the  renewal  who  also  reviewed  the  preceding 
grant.  Also,  when  a  renewal  proposal  is  submitted  to  NIH,  the 
primary  and  secondary  reviewers  in  th«  peer  review  group  are  pro¬ 
vided  copies  of  the  preceding  grant's  proposal  and  peer  review 
group  comments.  This  information  puts  them  in  a  better  position 
to  evaluate  the  progress  on  the  preceding  grant.  NSF  does  not 
provide  such  information  to  its  peer  reviewers. 

At  NSF,  the  turnover  in  cognizant  personnel  who  evaluate 
successive  grant  proposals  coupled  with  renewal  proposals  not 
having  the  objectives  and  progress  of  the  preceding  grant  specifi¬ 
cally  identified,  as  indicated  previously,  shows  that  the  scien¬ 
tific  performance  accountability  process  can  be  strengthened.  A 
clear,  easily  identified  presentation  of  the  preceding  grant 
progress  would  improve  proposal  evaluation  and  the  accountability 
process. 


1^/The  individual  responsible  for  making  the  sample  grant  award. 
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Evaluating  requests  for  new  project  support 


Researchers  who  submit  proposals  for  research  projects  which 
are  unrelated  to  their  preceding  grants  are  not  required  to  (1 ) 
include  in  the  proposal  a  progress  report  containing  the  results 
of  the  preceding  grant  or  (2)  specifically  identify  the  publica¬ 
tions  from  the  preceding  grant.  Also,  neither  NSF  or  NIH  requests 
peer  reviewers  to  specifically  evaluate  the  preceding  grant  per¬ 
formance.  Instead,  NSF  and  NIH  expect  that  peer  reviewers  will 
evaluate  preceding  grant  performance  as  a  part  of  the  researcher's 
general  track  record.  Fifteen  of  75  researchers  in  our  review 
submitted  subsequent  proposals  to  NSF  or  NIH  for  research  projects 
unrelated  to  their  preceding  grants. 

Agency  officials  contend  that  if  a  series  of  unrelated  grants 
had  produced  no  published  results,  it  would  negatively  affect  the 
evaluation  of  the  researcher's  proposals.  Both  NSF  and  NIH  pro¬ 
cedures  require  that  all  proposals  list  the  researcher’s  recent 
publications  as  a  general  indication  of  performance.  Relating 
publications  to  a  specific  grant  from  this  general  listing  is 
difficult,  especially  for  researchers  with  more  than  one  grant — 
about  two-thirds  of  those  in  our  review.  This  problem  is  further 
compounded  by  the  fact  that  many  publications  do  not  credit  the 
grant  from  which  they  came.  These  procedures  do  not  preclude  re¬ 
searchers  from  avoiding  accountability  on  unproductive  grants  by 
seeking  grants  for  new  projects. 

Our  review  provided  examples  of  how  specific  grant  perfor¬ 
mance  can  be  clouded  when  researchers  submit  proposals  for  proj¬ 
ects  unrelated  to  the  preceding  grants.  In  one  example,  an  NSF 
ad  hoc  approved  grant  for  $65,000  expired  in  December  1977,  but 
as  of  June  1979  results  had  yet  to  be  published  in  any  scientific 
journals.  Despite  this  lack  of  publications,  the  researcher 
received  a  subsequent  grant  in  March  1978  for  research  unrelated 
to  that  of  the  previous  grant.  Since  the  researcher  was  not 
required  to  discuss  the  previous  grant's  results  in  the  subsequent 
proposal,  this  lack  of  publications  was  not  apparent.  The  peer 
reviewers  for  the  subsequent  proposal  all  discounted  this  lack 
of  publications.  Ckie  peer  reviewer  was  aware  of  the  researcher’s 
general  track  record  and  felt  his  past  publications  had  been  sig¬ 
nificant.  The  remaining  peer  reviewers  said  that  knowing  of  this 
lack  of  publications  would  have  had  little  or  no  impact  since 
they  personally  place  evaluation  emphasis  on  scientific  merit 
rather  than  track  record. 

Another  NSF  ad  hoc  approved  grant  for  $17,000  expired  in 
December  1977  with  the  only  publication  from  the  grant  published 
in  1979.  There  is  also  a  question  regarding  the  researcher's  per¬ 
formance  under  the  preceding  grant.  His  proposal  objectives  in¬ 
cluded  an  analysis  of  a  broad  spectrum  or  group  of  soil  samples. 
The  researcher  even  requested  and  received  a  no-cost  extension 
through  December  1977  because  he  needed  more  time  to  analyze  more 
soils.  Still,  the  grant's  final  technical  report  discussed 
analysis  of  only  two  soils  as  opposed  to  the  broad  spectrum  or 
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group  originally  proposed.  A  subsequent  grant  unrelated  to  the 
preceding  grant,  was  awarded  by  the  program  officer  to  this 
researcher  effective  April  1,  1978 — despite  no  publications  from 
the  preceding  grant  at  the  time  of  the  renewal  and  questionable 
performance  under  the  preceding  grant.  Our  discussions  with  three 
of  the  peer  reviewers  for  the  subsequent  grant  showed  that  two  of 
them  were  not  aware  of  the  researcher's  performance  on  the  preced¬ 
ing  grant  while  the  third  could  not  recall.  All  three  stated  that 
having  results  of  the  preceding  grant  would  have  been  useful  to 
them  in  evaluating  the  subsequent  grant  proposal. 

These  examples  demonstrate  that  a  researcher  can  receive 
funding  for  a  project  unrelated  to  the  immediately  preceding  grant 
without  consideration  being  given  to  the  preceding  grant's  publi¬ 
cations  and/or  the  extent  to  which  originally  proposed  objectives 
were  attempted  or  accomplished.  This  problem  is  further  com¬ 
pounded  when  a  researcher  has  multiple  grants.  Since  publications 
are  not  related  to  specific  grants,  a  grant  which  resulted  in  no 
publications  may  not  come  to  light  and  be  considered  in  deciding 
to  fund  the  researcher  again.  To  provide  better  scientific  per¬ 
formance  accountability,  this  information  should  be  included  in 
new  project  proposals  for  consideration  by  the  peer  reviewers  as 
part  of  the  evaluation  of  the  new  project  proposal. 

Effect  of  peer  reviewers 
on  grant  award  decisions 

Peer  reviewer  evaluations  have  a  substantial  effect  on  grant 
awards  at  NSF  and  NIH.  Specifically,  at  NIH  we  found  that  peer 
reviewer  critiques  caused  on  renewal  proposal  to  be  declined  and 
three  others  to  be  approved  but  not  funded.  Additionally,  we 
found  19  instances  (7  at  NIH,  9  NSF  panel,  and  3  NSF  ad  hoc) 
where  peer  reviewer  comments  affected  grant  awards  in  terms  of 
either  proposed  research  objectives  being  eliminated,  or  budget 
and  time  reductions. 

Of  the  50  sample  researchers  seeking  renewal  grants  to  con¬ 
tinue  their  research  projects,  1  at  NIH  was  declined.  Peer 
reviewer  comments  cited  lack  of  progress  on  objectives,  defici¬ 
encies  in  technical  management,  and  failure  to  make  research 
results  available  to  the  scientific  community  as  the  reasons  for 
the  declination. 

In  the  three  cases,  which  were  approved  but  not  funded,  the 
peer  reviewers  approved  the  proposals  but  with  low  priority  which 
thereby  greatly  reduced  the  researchers’  chances  of  funding. 

The  problems  in  these  cases  were  (1)  researcher  needed  to  better 
focus  on  his  work,  (2)  research  was  having  little  impact  on  the 
rest  of  the  scientific  community,  and  (3)  research  had  come  to  a 
natural  end — objectives  remain  the  same  as  7  years  ago. 
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Peer  reviewers  also  affected  our  sample  grants  by  eliminating 
proposed  research  objectives  that  lacked  scientific  merit  and  by 
cutting  back  on  budget  and/or  time  requests.  In  six  instances 
(three  at  NIH,  one  NSF  panel  and  two  NSF  ad  hoc),  peer  reviewers 
felt  that  some  of  the  research  initially  proposed  was  question¬ 
able.  Subsequently,  NSF  and  NIH  did  not  fund  the  portions  of  the 
proposals  that  the  peer  reviewers  felt  were  questionable  and  re¬ 
duced  the  proposed  budget  to  reflect  the  eliminated  research. 

In  13  other  instances,  the  budget  and/or  time  requested  in  the 
proposal  was  reduced  based  on  the  comments  of  the  peer  reviewers. 
Four  of  these  were  at  NIH  and  the  other  nine  were  at  NSF  (eight 
panel  and  one  ad  hoc).  One  other  grant  at  NIH  and  two  at  NSF 
(one  panel  and  one  ad  hoc)  had  the  time  requested  increased  be¬ 
cause  peer  reviewers  believed  the  research  worthwhile  but  that 
the  researcher  needed  more  time  to  accomplish  it. 

Our  contacts  with  peer  reviewers  suggest  that  their  knowledge 
of  sample  researchers'  accomplishments  contributed  to  their  re¬ 
views.  Fifty-four  of  58  (93  percent)  peer  reviewers  contacted 
were  aware  of  the  researcher's  track  record,  and  had  previously 
read  the  researcher's  publications.  Additionally,  about  three- 
fourths  of  the  peer  reviewers  knew  the  researchers  personally  at 
the  time  they  critiqued  the  proposals. 

THE  AWARD  DECISION 


The  second  major  part  of  the  accountability  process  for  basic 
research  grants  at  NSF  and  NIH  is  making  the  award  decision.  The 
basic  decision  regarding  which  proposals  will  be  funded  as  grants 
at  NSF  is  made  by  the  program  officers.  At  NIH,  this  decision  is 
made  by  NIH  program  officials  after  the  proposals  have  been  ap¬ 
proved  by  the  peer  review  group  and  National  Advisory  Council. 

NIH  peer  reviewers  are  responsible  for  determining  the  scientific 
merit  of  proposals  but  not  funding  decisions. 

Relying  on  peer  reviewers  varies 

Both  NIH,  and  NSF  to  a  lesser  extent,  rely  on  peer  reviewer 
advice  in  awarding  research  grants.  NSF  program  officers  play  a 
much  larger  part  in  the  award  decision  than  their  counterparts 
at  NIH.  Also,  NIH  does  not  focus  the  award  process  on  one  person. 

At  NIH,  deciding  whether  a  proposal  has  scientific  merit, 
and  determining  the  amount  and  duration  of  the  award,  if  made, 
initially  rests  with  the  peer  review  groups.  These  decisions  are 
reviewed  by  the  National  Advisory  Council  of  each  Institute.  1/ 

The  NIH  program  officers  then  select  approved  proposals  from  the 
Council-approved  list  for  funding. 


1/The  peer  review  group  makes  a  recommendation  to  the  National  Ad¬ 
visory  Council.  The  Council  concurs  with  these  recommendations 
in  the  vast  majority  of  cases. 
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The  peer  review  process  culminates  in  priority  scores  that 
are  the  basis  for  funding  each  approved  proposal.  Since  peer 
reviewers  make  these  determinations,  NIH  funding  procedures  rely 
heavily  on  peer  evaluations. 

NIH  health  science  administrators  (program  officers) ,  how¬ 
ever,  may  identify  proposals,  which  were  approved  by  the  peer 
review  group,  having  special  program  relevance  that  warrant  fund¬ 
ing  without  regard  to  priority.  Some  of  these  proposals  would 
not  be  funded  on  the  basis  of  the  priority  scores  alone.  Health 
science  administrators  indicated  that  proposals  are  only  infre¬ 
quently  funded  without  regard  to  priority.  Estimates  ranged  from 
less  than  1  percent  to  3  percent  of  all  proposals  funded. 

The  award  of  two  grants  demonstrates  how  this  procedure 
works.  The  Institute  funding  the  grants  awarded  117  grants  dur¬ 
ing  fiscal  year  1978  to  proposals  that  had  priority  scores  rang¬ 
ing  from  103  (the  best)  to  361  (the  worst).  Two  proposals  having 
program  relevance,  however,  were  also  funded  without  reference 
to  their  scores  of  268  and  361.  These  scores  were  below  the  254 
priority  score  of  the  last  proposal  funded  on  the  basis  of  prior¬ 
ity  scores  alone.  These  2  proposals  of  117  proposals  funded 
constituted  1.7  percent. 

The  NSF  program  officer  makes  the  decision,  with  the  advice 
of  the  peer  reviewers,  as  to  whether  the  proposal  has  scientific 
merit,  whether  or  not  it  should  be  funded,  the  amount  and  duration 
of  the  award.  Prior  to  making  the  scientific  merit  and  funding 
decisions,  the  NSF  program  officer  also  (1)  performs  the  initial 
proposal  review  for  relevance,  general  merit  and  funds  availabil¬ 
ity,  (2)  selects  the  ad  hoc  peer  reviewers  to  evaluate  the  propo¬ 
sal,  (3)  selects  the  panel  peer  reviewers,  if  a  panel  is  used, 

(4)  evaluates  the  peer  review  comments,  and  (5)  makes  a  site 
visit,  if  necessary. 

At  NSF,  peer  reviewers'  comments  are  strictly  advisory  to 
the  program  officer  and  the  degree  to  which  the  officers  rely  on 
peer  reviewer  comments  is  less  clear.  Twenty-five  NSF  program 
officers,  interviewed  in  1978  by  congressional  staff  members, 
said  that  their  approach  to  weighing  a  given  peer  reviewer’s  com¬ 
ments  against  their  own  opinion  depends  on  their  knowledge  in  the 
field  of  the  proposal  and  on  how  strongly  they  hold  their  opin¬ 
ions.  Several  added  that  negative  peer  comments  almost  always  re¬ 
sult  in  declinations  regardless  of  the  program  officer's  opinion. 

One  example,  however,  demonstrates  the  flexibility  that  an 
NSF  program  officer  has  in  dealing  with  peer  reviewer  comments. 

Ten  ad  hoc  peer  reviewers  were  solicited  to  evaluate  the  proposal. 
Their  ratings  were:  one  excellent,  four  very  good,  one  good,  two 
fair,  one  only  provided  comments  with  no  rating  and  one  disquali¬ 
fied  himself  because  he  felt  he  could  not  be  an  impartial  re¬ 
viewer.  The  overall  rating  was  between  good  and  very  good.  The 
main  problem  of  the  three  reviewers  who  rated  the  proposal  as  good 
and  fair  was  that  they  did  not  believe  the  research  had  potential 
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for  significant  scientific  advancement.  The  program  officer, 
however,  believed  the  research  had  a  potential  payoff.  He,  there¬ 
fore,  omitted  the  two  fair  ratings  in  arriving  at  the  final  rat¬ 
ing  of  very  good — he  did  not  omit  the  good  rating.  He  also  did 
not  omit  two  of  the  very  good  ratings  with  accompanying  comments 
termed  by  the  program  officer  as  "brief,  friendly  and  fairly 
empty."  The  grant  was  awarded.  The  program  officer  who  made  the 
award  was  no  longer  at  NSF  at  the  time  of  our  review. 

When  NSF  program  officers  disagree  with  peer  reviewers  on 
significant  points,  they  should  justify  and  document  their  dis¬ 
agreements.  Reasons  for  disagreement  cited  by  program  officers 
included  such  reasons  as  knowledge  of  prior  grants,  perspective 
gained  through  seeing  a  wide  range  of  proposals,  and  knowledge  of 
the  rating  styles  (hard  vs.  easy)  of  individual  reviewers.  In 
one  sample  grant,  a  program  officer  disagreed  with  the  peer  re¬ 
viewer's  assertion  that  the  grant's  proposal  was  too  ambitious  or 
grandiose.  Since  the  program  officer  did  not  act  on  this  criti¬ 
cism,  the  grant  was  awarded  as  requested.  But  in  this  case  the 
researcher  later  requested  a  grant  extension  because  progress  was 
not  on  schedule. 

We  asked  all  program  officials  we  interviewed  at  NSF  and  NIH 
if,  prior  to  making  their  decisions  on  all  proposals,  they  rou¬ 
tinely  contact  researchers  regarding  negative  peer  reviewer 
comments  (see  table  5). 


Table  5 


Do  Program  Officials  Routinely 


Negative  Peer  Review  Comments  on 

All  Proposal 

NIH 

executive 

secretary 

s  They  Review? 

NIH 

health  science 
administrator 

NSF 

Total 

Yes 

3 

1 

6 

10 

No 

7 

14 

18 

39 

No  response 

3 

7 

11 

21 

‘ 

Total 

13 

22 

35 

70 

' 

!  responses 

indicate  that  program  officials 

usually  do  not 

contact  researchers  regarding  negative  peer  comments,  but  exer¬ 
cise  their  own  judgment  in  assessing  the  comments. 


Because  of  the  flexibility,  judgment,  and  decisionmaking 
authority  exercised  by  NSF  program  officers,  we  asked  the 
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cognizant  program  officers  1/  if  they  could  have  been  peer 
reviewers,  i.e.,  were  sufficiently  knowledgeable,  for  the  grants 
we  reviewed.  Table  6  gives  the  results. 

Table  6 


Cognizant  Program  Officers  with  Enough 


Knowledge  to  Have  Been  a 

Peer  Reviewer 

on  the  Sample  Grant 

Proposal 

NIH 

NSF 

Tote 

Yes 

10 

17 

27 

No 

9 

10 

19 

No  response 

1 

_1 

2 

20 

28 

48 

The  NSF  responses  indicate  that  a  majority  of  the  program 
officers  felt  that  they  had  enough  knowledge  to  have  been  peer 
reviewers  on  the  sample  grants.  The  1977  NSF  study  discussed 
earlier  (on  p.  18)  concluded,  however,  that  peer  review  ratings 
are  the  most  important  determinant  of  the  program  officer's  deci¬ 
sion.  This  study  compared  peer  ratings  to  awards  made  and  showed 
92  percent  of  the  proposals  receiving  a  comparatively  high  rating 
were  funded.  Conversely,  only  10  percent  of  those  receiving  a 
low  rating  were  funded.  About  half  of  those  with  middle  range 
ratings  were  funded.  The  study  concluded  that  NSF  program  offi¬ 
cers  rely  "heavily"  on  peer  rating. 

While  NSF  program  officers  rely  on  peer  reviewer  advice,  it 
should  be  noted  that  in  most  instances  peer  reviewers  do  not  pri¬ 
oritize  the  proposals  reviewed  and  do  not  recommend  specific  dol¬ 
lar  amounts  for  the  award.  NSF  program  officers  make  these  deter¬ 
minations  . 

As  indicated  earlier,  no  single  evaluation  process  exists  at 
NSF.  As  such,  the  role  of  the  peer  reviewers  also  varies.  Some 
NSF  panel  peer  reviewers  do  prioritize  the  proposals  reviewed, 
some  do  not,  and  none  of  the  proposals  that  are  solely  ad  hoc 
reviewed  are  prioritized. 

Regarding  the  amount  of  the  grant,  NSF  peer  reviewers  are 
requested  to  comment  on  the  proposed  budget.  Their  comments, 
however,  are  usually  general  in  nature,  i.e.,  budgets  appear  rea¬ 
sonable  or  excessive.  Only  rarely,  will  an  NSF  peer  reviewer 
specifically  comment  on  how  much  should  be  deleted  or  added  to  a 
proposed  budget — usually  leaving  this  determination  to  the  program 
officer.  Most  of  the  25  program  officers  interviewed  in  1978  by 


1_/The  individual  responsible  for  making  the  sample  grant  award. 
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congressional  staff  members  said  that  they  learned  about  budget 
evaluation  on  the  job.  About  half  had  prior  research  experience 
useful  for  budget  evaluations.  (Evaluations  generally  entail  see¬ 
ing  if  the  amount  of  money  requested  is  what  is  needed  to  do  the 
research.  In  most  programs,  the  research  is  similar  enough  that 
comparisons  can  be  made  among  proposals  and  funded  grants.)  The 
program  officers  also  reported  that  experience  has  taught  them 
what  certain  items  cost,  and  five  added  that  they  eliminate  desir¬ 
able  but  non-essential  items  in  order  to  fund  as  many  scientists 
as  possible.  Program  officers  we  interviewed  made  similar 
comments . 

Documenting  results  of  peer 
review  panel  deliberations 

The  NSF  peer  review  panels  function  similarly  to  that  of 
the  NIH  peer  review  groups,  i.e.,  a  number  of  peer  reviewers  meet 
to  decide  from  a  group  of  proposals  those  most  deserving  of  sup¬ 
port.  The  results  of  the  NIH  peer  review  group  deliberations  are 
clearly  and  uniformly  documented.  The  results  of  the  NSF  panel 
deliberations,  however,  are  not. 

Twenty-one  NSF  panel  grants  were  included  in  our  review.  Our 
analysis  of  the  peer  review  comments  for  these  grants  disclosed 
that  for  nine  of  these  grants  (about  43  percent)  there  was  no 
summary  of  the  panel  meeting.  Fourteen  of  these  21  grants  were 
renewed.  In  5  of  these  14  instances  (about  36  percent)  there 
was  also  no  summary  of  the  panel  meeting.  In  many  cases  where 
there  was  evidence  of  the  panel  meeting,  it  was  one  paragraph  or 
simply  handwritten  notes.  In  all  cases  there  was  a  panel  rating. 

At  NIH,  at  the  conclusion  of  the  peer  review  group  meeting, 
the  executive  secretary  prepares  a  summary  statement  which  sum¬ 
marizes  the  peer  reviewers'  critiques  and  recommendations.  This 
statement  represents  the  group’s  decision  and  does  not  divulge 
the  views  of  individual  peer  reviewers.  The  summary  statement 
is  based  on  the  written  evaluation  prepared,  in  an  almost  iden¬ 
tical  format  to  the  summary  statement,  by  the  reviewers  prior  to 
the  meeting  and  the  deliberations  at  the  meeting.  In  addition 
to  a  recommendation  and  priority  score,  the  summary  statement 
includes  a  resume  of  the  reasons  for  the  recommendation,  a  de¬ 
scription  of  the  research,  a  critique  of  the  research,  a  comment 
on  the  researcher's  qualifications  and  competence,  a  comment  on 
the  resources  and  environment  available  to  the  researcher,  anc  •» 
budget  evaluation.  There  was  a  summary  statement  for  all  NIH 
grants  we  reviewed. 

Releasing  peer  review  comments  to  researchers 

NSF  and  NIH  policies  differ  on  releasing  peer  review  comments 
to  researchers.  NIH  automatically  releases  the  comments,  while 
NSF  only  releases  them  upon  a  researcher's  request.  These  com¬ 
ments  can  be  useful  to  researchers  in  conducting  research  or  im¬ 
proving  declined  proposals  for  resubmission  and  re-evaluation. 
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For  example,  peer  comments  address  strengths  and  weaknesses  in 
approach,  feasibility,  and  appropriateness  of  procedures,  facil¬ 
ities,  equipment,  etc. 

NIH  automatically  sends  summary  statements  with  priority 
scores  to  researchers  after  a  Council's  review.  Researchers  may 
also  request  a  summary  statement  copy  without  its  priority  score 
prior  to  the  Council's  review.  One  NSF  directorate  is  automati¬ 
cally  providing  researchers  with  peer  comments  on  an  experimental 
basis.  NSF  plans  to  evaluate  the  experiment  to  determine  the 
benefits  of  automatically  providing  researchers  with  peer  com¬ 
ments.  The  process  has  been  well-received  by  researchers,  accord¬ 
ing  to  NSF  officials.  We  believe  NSF  should  establish  a  policy 
of  automatically  providing  researchers  with  peer  review  comments. 

About  two-thirds  of  the  researchers  included  in  our  review 
did  not  receive  the  peer  review  comments  for  their  grants.  Of 
those  receiving  the  comments,  more  than  half  felt  they  were  use¬ 
ful  for  such  reasons  as  changing  the  planned  research,  improving 
proposal  preparation  techniques,  and  obtaining  alternate  funding. 
We  analyzed  the  peer  review  comments  for  those  researchers  who 
did  not  receive  them.  In  about  one-third  of  these  cases,  we 
believe  these  comments  would  have  been  helpful  to  the  researcher 
because  they  offered  suggestions  and/or  alternative  approaches. 

Failure  to  forward  peer  comments  to  researchers  can  adversely 
affect  research.  One  researcher,  for  example,  was  unaware  of  why 
a  requested  piece  of  equipment  was  deleted  from  his  proposed  bud¬ 
get.  NIH  peer  reviewers  erroneously  thought  he  would  have  access 
to  this  equipment  at  his  university.  He  was  unable  to  obtain 
access  and  thus  did  not  complete  a  portion  of  his  research.  If 
he  had  been  aware  of  the  peers'  rationale  for  deleting  this  equip¬ 
ment,  he  could  have  requested  supplemental  funding  for  the  neces¬ 
sary  equipment.  NIH  has  changed  its  policy  since  this  example 
occurred  so  that  peer  reviewer  comments  are  now  automatically 
provided  to  researchers. 

In  a  second  instance  (NSF  panel),  a  peer  reviewer  commented 
that  he  thought  some  experiments  might  be  in  jeopardy  because  of 
certain  factors  which  he  stated.  The  researcher  did  subsequently 
have  some  problems  with  these  experiments  for  precisely  the  rea¬ 
sons  stated  by  the  peer  reviewer.  He  never  saw  the  peer  review 
comments.  The  program  officer  acknowledged  that  in  this  instance 
the  comments  should  have  been  sent  to  the  researcher. 

In  another  NSF  panel  grant,  a  peer  reviewer  suggested  that 
the  researcher  should  widen  his  area  of  research.  The  researcher, 
who  did  not  receive  his  peer  review  comments,  told  us  that  this 
peer  reviewer's  suggestion  and  comment  were  quite  accurate  and 
that  it  would  have  been  helpful  to  him  to  have  seen  this  comment 
earlier . 

In  addition  to  contributing  to  the  conduct  of  funded  re¬ 
search,  peer  comments  are  useful  to  researchers  whose  proposals 
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have  been  declined  *  ,  *ing.  Several  program  officials  said  that 
declined  proposals  can  oe  revised  to  correct  shortcomings  identi¬ 
fied  by  peer  reviewers  and  resubmitted  for  reevaluation.  This 
simple  procedure  assists  researchers  in  obtaining  grants  and  also 
alleviates  the  need  for  formal  rebuttal  procedures  to  resolve  any 
disagreements  between  researchers,  peer  reviewers,  or  agency 
officials.  We  found  one  proposal  that  was  awarded  after  the  re¬ 
searcher  had  revised  an  earlier  disapproved  proposal  to  correct 
problems  identified  in  peer  comments.  Peer  reviewers  noted  that 
the  "revised  proposal  and  the  information  provided  at  the  site 
visit  addressed  the  previous  criticisms,  and  progress  has  been 
made  on  strengthening  a  number  of  aspects  of  the  project  which 
were  quite  deficient." 

In  commenting  on  a  draft  of  this  report,  NSF  stated  that  it 
will  now  make  universal  the  routine  forwarding  of  peer  review 
comments  to  researchers. 

MONITORING  THE  RESEARCH 

Monitoring  is  a  process  whereby  the  management  and  perform¬ 
ance  of  a  research  grant  are  continuously  reviewed  through  the 
collection  and  assessment  of  information  gathered  from  various 
reports,  site  visits,  and  other  sources.  Agency  program  officers 
are  responsible  for  monitoring  federally  funded  grants  to  assure 
that  the  terms  of  the  grant  are  carried  out.  Agencies'  grant 
monitoring  guidance  varies  and,  consequently,  the  types  and 
degree  of  research  grant  monitoring  varies  between  NSF  and  NIH, 
and  within  the  two.  Also,  both  agencies'  instructions  require 
that  the  universities  receiving  the  grants  take  an  active  role 
in  grant  monitoring. 

Progress  reports 

NSF  and  NIH  program  officials  are  responsible  for  monitoring 
scientific  progress.  NSF  program  officers  are  supposed  to  review 
progress  and  final  technical  reports  and  may  make  site  visits. 

NIH  program  officials  similarly  are  responsible  for  reviewing 
annual  and  final  progress  reports  and  may  make  site  visits. 

Our  review  showed  that  NSF  and  NIH  program  officials  primar¬ 
ily  rely  on  progress  reports  to  monitor  scientific  progress.  NSF 
requires  annual  progress  reports  on  all  grants  lasting  2  or  more 
years.  NSF  currently  requires  that  its  progress  reports  contains 
(1)  a  summary  of  overall  progress  to  date  and  its  relationship 
to  proposed  objectives,  (2)  an  indication  of  any  problems  and 
favorable  or  unusual  developments,  (3)  a  summary  of  work  to  be 
performed  during  the  succeeding  budget  period,  and  (4)  other  per¬ 
tinent  information  to  the  type  of  project  or  information  specified 
in  the  grant's  terms  and  conditions. 

NIH  also  requires  progress  reports  annually  in  conjunction 
with  requests  for  continued  support.  NIH  currently  requires  that 
its  progress  reports  cover  (1)  research  objectives  (overall  and 
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current  year),  (2)  results,  to  include  their  significance,  rela¬ 
tionships  to  objectives,  changes  in  direction,  negative  results, 
and  technical  problems,  (3)  a  description  of  the  study's  signifi¬ 
cance  to  health  problems,  and  (4)  objectives  for  the  coming  year. 

Both  NSF  and  NIH  funded  researchers  appear  to  have  complied 
with  progress  report  requirements.  NSF  funded  researchers,  in¬ 
cluded  in  our  review,  that  had  grants  with  a  duration  exceeding 
2  years  submitted  progress  reports  as  required.  For  the  progress 
reports  we  analyzed,  all  NIH  funded  researchers  also  submitted 
progress  reports  as  required  with  relatively  few  omissions  in 
required  content. 

Both  NSF  and  NIH  require  program  officials  to  review  progress 
reports.  Neither,  however,  specifies  how  these  reviews  will  be 
conducted.  Program  officials  varied  in  conducting  these  reviews. 
They  said  that  they  receive  and  read  progress  reports.  However, 
only  5  of  the  35  NSF  program  officers  interviewed  said  that  they 
compare  progress  reports  to  originally  proposed  research  objec¬ 
tives  to  evaluate  specific  progress  toward  their  accomplishment, 
while  about  half  (10  of  22)  of  the  NIH  program  officials  inter¬ 
viewed  made  such  a  comparison.  Additionally,  for  7  of  the  25  NIH 
grants  reviewed,  the  NIH  program  officials  prepared  a  progress 
report  review  check  sheet.  This  check  sheet  requires  the  program 
official  to  state  if  (1)  there  were  publications,  (2)  the  report 
provides  the  required  information,  (3)  the  project  has  changed 
direction,  and  (4)  further  administrative  action  is  required. 

The  check  sheet  also  requires  the  program  officer  to  explain  the 
highlights  of  the  progress  to  date.  Notwithstanding  these  check 
sheets,  we  did  not  find  that  any  grant  was  changed  as  a  result  of 
a  program  officer's  review  of  progress  reports. 

Most  program  officials  said  that  progress  reports  are  gener¬ 
ally  useful  in  keeping  them  informed  of  a  researcher's  progress, 
in  monitoring  progress,  or  in  making  a  decision  to  renew  funding 
of  a  research  project.  Typical  program  official  comments  on  prog¬ 
ress  reports  are  as  follows: 

— Researchers  are  not  pressed  for  progress  reports  because 
he  does  not  feel  a  great  deal  of  progress  can  be  made  in 
one  year.  He  prefers  that  researchers  delay  reporting 
until  significant  results  are  achieved  rather  than  submit¬ 
ting  reports  because  of  agency  requirements. 

— Progress  reports  are  not  compared  to  originally  proposed 
research  objectives.  He  evaluates  productivity  through 
publications  and  the  detailing  of  events  in  progress 
reports  rather  than  the  degree  to  which  original  research 
objectives  are  completed. 

— Progress  reports  are  compared  to  original  objectives. 
Consequently,  progress  reports  are  useful  in  evaluating 
progress  toward  original  objective  accomplishment. 
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— Progress  reports  are  important  because  they  discuss  prog¬ 
ress  or  lack  thereof.  He  added  that  if  progress  has  not 
been  made,  it  is  important  that  researchers  discuss  how 
problems  will  be  overcome  in  the  coming  period. 

In  addition  to  progress  reports,  program  officials  also  ob¬ 
tain  information  on  grant  progress  through  informal  means  such  as 
site  visits,  telephone  conversations,  and  contacts  at  professional 
meetings.  Such  contact  occurred  on  about  three-fourths  of  the 
grants  we  reviewed. 

Site  visits 


Both  NSF  and  NIH  program  officials  may  make  site  visits. 
Neither,  however,  specifies  when  such  visits  should  occur.  As 
with  progress  reports,  program  officials'  approaches  to  making 
site  visits  vary.  Fifteen  site  visits  were  made  to  the  research¬ 
ers  in  our  review.  Two  were  by  NIH  officials  and  13  by  NSF 
officials.  Of  these  15  visits,  4  were  made  as  part  of  the  award 
decision  process  as  opposed  to  during  the  grant  period. 

Reasons  for  making  site  visits  also  varied.  The  largest 
number  were  not  made  to  specifically  review  grant  progress  but 
were  made  in  conjunction  with  other  purposes,  such  as  attending 
professional  meetings  in  the  same  geographical  area.  NSF  pointed 
out  that  contacts  with  researchers  during  professional  meetings 
is  a  cost-effective  way  of  accomplishing  the  objectives  of  a  site 
visit  in  that  both  staff  time  and  travel  funds  are  conserved. 

Other  reasons  included  the  large  size  of  the  award,  the  need  for 
additional  information,  and  the  nature  of  the  research  activity 
(theoretical  versus  experimental  research).  Typical  program 
official  comments  on  making  site  visits  follows: 

— He  visits  researchers  having  the  most  activity  and  strong¬ 
est  potential  for  significant  results.  He  added  these 
visits  are  not  made  as  a  result  of  indications  of  problems, 
but  rather  to  check  research  progress,  attend  professional 
meetings  in  the  same  geographical  area,  and  possibly 
visit  potential  grantees  to  make  them  aware  of  the  agency's 
funding  capabilities. 

— Site  visits  are  made  to  grantees  having  large  dollar 
awards.  He  added  that  visits  are  made  to  grantees  at 
laboratories  having  equipment  and  experiments  that  can  be 
seen,  discussed  and  evaluated.  Site  visits  are  not  made 
to  grantees  performing  essentially  theoretical  research. 

--Site  visits  are  made  to  obtain  assurances  that  research 
is  being  conducted  with  a  potential  for  significant  re¬ 
sults.  Consequently,  he  does  not  use  a  checklist  ap¬ 
proach  in  relating  research  to  originally  proposed  goals. 
His  concern  instead  is  with  the  significance  of  the  re¬ 
search  actually  being  conducted. 
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Peer  reviewers  *  role 


Progress  on  research  grants  is  not  uniformly  monitored.  The 
uses  program  officials  made  of  progress  reports  varied  from  just 
reading  them  to  comparing  them  with  original  research  objectives 
to  evaluate  progress  toward  accomplishment.  Uses  of  other  mon¬ 
itoring  techniques  such  as  site  visits  were  similarly  varied. 
Several  program  officials  said  that  scientific  progress  is  not 
closely  monitored  during  the  grant  period.  Instead,  they  rely 
on  peer  reviewers  to  evaluate  research  results  when  a  researcher 
seeks  a  subsequent  grant  to  continue  a  line  of  research.  If  peer 
reviewers  determine  that  a  researcher  has  been  unproductive,  sub¬ 
sequent  grants  may  not  be  awarded  to  continue  the  project.  How¬ 
ever,  19  percent  of  the  researchers  in  our  sample  did  not  seek 
another  grant  from  NSF  or  NIH. 

Several  program  officials  said  that  relying  on  peer  reviews 
is  a  practical  solution  to  their  lack  of  time  for  closely  moni¬ 
toring  grants.  Similar  thoughts  were  expressed  by  25  NSF  program 
officers  interviewed  in  1978  by  congressional  staff  members. 

About  three-quarters  of  these  program  officers  said  that  they  do 
less  monitoring  than  they  believe  would  be  desirable.  They 
attributed  this  to  limited  travel  money  and  time. 

Che  program  official  added  that  adequate  monitoring  requires 
the  expertise  of  peer  reviewers.  He  said  that  program  officials 
often  do  not  have  this  expertise.  About  40  percent  of  the  program 
officials  we  interviewed  said  they  would  not  have  had  the  exper¬ 
tise  necessary  to  serve  as  peer  reviewers  for  sample  grants. 

Monitoring  implies  termination  for  lack  of  progress.  Both 
NSF  and  NIH  have  grant  termination  policies  which  state  that  a 
grant  can  be  terminated  if  a  grantee  fails  to  comply  with  the 
terms  and  conditions  of  the  grant.  Both  policies  also  state  that 
correction  of  deficiencies  is  preferable  whenever  practicable. 

Both  provide  for  notification  of  deficiencies,  and  time  for 
corrective  actions  prior  to  termination. 

Officials  at  both  NSF  and  NIH  sail  that  grants  are  almost 
never  terminated  for  lack  of  productivity.  An  NSF  official  gave 
several  reasons  for  why  grants  are  not  terminated.  First,  NSF 
attempts  to  solve  problems  before  taking  such  actions.  Second, 
standard  grants  normally  last  for  1  or  2  years,  and  this  is  not 
sufficient  time  to  enable  the  agency  to  assess  lack  of  progress 
with  sufficient  conviction  to  terminate  a  grant.  Third,  re¬ 
searchers  are  only  expected  to  put  forth  their  "best  effort"  in 
accomplishing  research  goals.  And  lastly,  the  taxpayer’s  inter¬ 
ests  are  best  served  by  bringing  a  project  to  an  orderly  close 
rather  than  "chopping  off"  the  project. 


Universities  do  not  monitor 
scientific  progress  ” 


The  NSF  Grant  Policy  Manual  requires  that  grantees  (univer¬ 
sities)  monitor  the  performance  of  research  projects  to  assure 
adherence  to  (1)  such  performance  goals,  time  schedules,  or  other 
requirements  as  may  be  appropriate  to  the  project  and  the  terms  of 
the  grant?  and  (2)  sound  management  practices  and  organizational 
policies.  Hie  manual  also  prescribes  the  ways  in  which  grantees 
should  carry  out  proper  monitoring  and  approval  in  advance  of  any 
action  that  would  result  in  either  performance  changes  or  modifi¬ 
cation  of  an  NSF  grant. 

NIH  requires  that  universities  be  responsible  and  accountable 
for  the  performance  of  grant-supported  activity.  NIH  places 
maximum  reliance  on  its  grantees'  (universities)  controls  and  re¬ 
quirements.  NIH  expects  that  its  grantees  will  exercise  sound 
management  practices  in  carrying  out  the  terms  and  conditions  of 
research  grants. 

The  universities  we  visited  do  not  monitor  scientific  prog¬ 
ress  despite  NSF's  and  NIH's  policy  that  they  should.  NSF  policy 
requires  universities  to  insure  that  researchers  adhere  to  per¬ 
formance  goals,  schedules,  project  and  grant  requirements,  and 
sound  management  and  organizational  policies.  NIH  policy  states 
that  universities  assume  legal  and  financial  responsibility  and 
accountability  both  for  the  awarded  funds  and  for  performance  of 
the  grant-supported  activity.  Only  1  of  the  7  5  researchers  we 
interviewed  said  that  the  university  monitored  grant  progress. 
Several  researchers  said  that  the  necessary  expertise  was  not 
present  at  their  university  to  monitor  their  research  activities. 

EVALUATING  THE  RESEARCH 


Evaluating  the  results  of  basic  research  grants,  as  in 
evaluating  the  proposals  which  result  in  the  grants,  is  carried 
out  almost  totally  within  the  scientific  community.  In  evaluat¬ 
ing  research  results,  as  in  proposal  evaluations,  both  NSF  and 
NIH  rely  on  peer  review — but  of  a  somewhat  different  form  than 
that  used  to  evaluate  research  proposals.  Both  agencies  also  re¬ 
quire  researchers  to  submit  final  technical  reports  on  completed 
projects . 

Outside  of  peer  review 

Neither  NSF  nor  NIH  routinely  evaluate  the  results  of  re¬ 
search  grants.  Outside  of  the  peer  review  process  very  little 
evaluation  of  research  results  is  conducted.  If  researchers  do 
not  seek  additional  support,  however,  their  performance  on  the 
preceding  grant  is  generally  not  evaluated.  Fourteen  of  the  75 
researchers  included  in  our  review  (19  percent)  did  not  seek  addi¬ 
tional  support.  Three  (12  percent  of  the  25  NIH  grants  reviewed) 
of  these  14  were  NIH  funded  while  11  (22  percent  of  the  50  NSF 
grants  reviewed)  were  funded  by  NSF.  These  NSF  statistics  are 
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somewhat  different  than  those  provided  by  NSF  officials  during 
FY  1980  Senate  appropriations  hearings.  For  one  typical  basic 
research  program,  NSF  indicated  that  more  than  90  percent  of  the 
researchers  who  completed  their  grants  during  FY  1978  (the  same 
time  period  as  our  sample  grants)  submitted  new  proposals  for 
further  research. 

Agency  officials  can  evaluate  research  results  for  all  grants 
through  final  reports  and  publications.  Both  NSF  and  NIH  require 
final  project  reports  which  basically  summarize  progress  toward 
accomplishing  original  research  objectives,  significant  results 
actually  achieved,  published  or  planned  publications  resulting 
from  the  grant,  and  information  on  project  personnel.  NSF  re¬ 
quires  a  final  project  report  at  each  grant's  termination.  In 
contrast,  NIH  only  requires  a  terminal  report  at  the  end  of  a 
project  which  may  run  over  several  successive  grants.  Informa¬ 
tion  on  grant  progress  and  results,  however,  is  available  to  NIH 
through  annual  interim  progress  reports  and  renewal  proposal  prog¬ 
ress  reports.  NSF  and  NIH  also  require  that  researchers  submit 
publication  reprints  and  information  on  such  items  as  inventions, 
patents,  or  material  specifically  required  by  the  grant  instru¬ 
ment.  This  information  is  available  for  evaluation  by  program 
officials  and  is  used  to  inform  the  Congress  and  the  general 
public  about  grant  results. 

With  two  exceptions,  the  researchers  in  our  review  submitted 
final  reports  as  required  for  all  of  our  sample  grants.  Program 
officials  said  that  they  read  these  reports,  but  only  21  percent 
said  they  actually  compared  these  reports  to  the  originally  pro¬ 
posed  research  objectives  to  evaluate  progress  toward  their 
accomplishment.  These  officials  said  that  they  used  final  reports 
for  such  purposes  as  evaluating  a  researcher's  performance  and 
closing  out  a  grant.  A  few  said  that  these  reports  are  only 
submitted  to  fulfill  an  agency  requirement  and  that  they  actually 
serve  little  purpose.  Most  researchers  felt  final  reports  were 
merely  a  necessary  evil.  However,  for  researchers  who  do  not 
seek  a  renewal  grant,  the  final  report  may  provide  the  best 
disclosure  of  a  researcher's  performance  under  a  specific  grant. 

Publications  are  also  submitted  to  program  officials  as  evi¬ 
dence  of  research  performance.  Eighty-three  percent  of  the  75 
researchers  included  in  our  review  published  some  results  of  the 
grants  we  reviewed  in  peer  reviewed  journals — the  method  generally 
regarded  as  the  most  important  measure  of  a  researcher's  perform¬ 
ance.  Although  program  officials  said  that  they  received  the  pub¬ 
lications,  only  20  percent  of  the  NIH  officials  and  32  percent  of 
the  NSF  officials  said  that  they  compared  these  publications  to 
originally  proposed  research  objectives  to  evaluate  their  accom¬ 
plishment.  A  few  indicated  they  did  not  even  read  them.  One  pro¬ 
gram  official  summarized  a  common  view  by  saying  that  assessing 
results  in  relationship  to  original  objectives  is  not  appropriate 
since  researchers  are  awarded  grants,  not  contracts.  The  official 
felt  that  researchers  thus  should  only  be  held  accountable  for 
making  an  honest  effort  to  accomplish  original  objectives  and  that 
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actual  results  should  be  evaluated  for  their  own  significance 
rather  than  for  their  relationship  to  original  objectives. 

Evaluating  results  through  final  reports  and  publications  by 
program  officials  has  limited  immediate  value.  Forty  percent  of 
the  program  officers  (45  percent  at  NIH  and  36  percent  at  NSF)  we 
interviewed  stated  that  they  could  not  have  been  peer  reviewers 
for  the  grant  in  question.  These  officials  do  not  have  the  exper¬ 
tise  to  evaluate  some  grant  results.  Additionally,  final  reports 
are  not  required  until  after  a  grant's  or  project's  expiration. 
Similarly,  a  grant's  results  frequently  are  not  published  until 
after  it  has  expired.  Consequently,  as  one  program  official 
said,  evaluation  of  "water  under  the  bridge"  has  little  usefulness 
in  monitoring  or  policing  a  grant.  Typical  program  official 
comments  on  final  reports  and  publications  are  as  follows: 

— Final  reports  are  useful  in  providing  general  informa¬ 
tion  about  a  grant.  Additionally,  they  alert  [the  program 
official]  to  anything  out  of  the  ordinary  from  his  basic 
understanding  of  the  grant. 

— Final  reports  are  read  and  compared  to  research  objectives. 
However,  these  reports  are  mostly  just  a  record  and  re¬ 
prints  of  publications  are  more  useful. 

— Publications  are  read  and  compared  to  research  objectives 
because  sometimes  they  do  not  resemble  the  grant  progress 
as  summarized  in  a  researcher's  progress  reports. 

— Publications  are  useful  in  evaluating  productivity,  but  it 
is  not  reasonable  to  assume  they  should  relate  to  all  re¬ 
search  objectives  because  many  objectives  do  not  work  out. 

— Publications  are  read  to  ascertain  progress  made  toward 
research  objectives.  Nb  attempt  is  made  to  specifically 
match  publications  to  objectives  because  these  objectives 
only  provide  an  indication  of  where  the  researcher  intends 
to  go  during  a  grant  period.  Additionally,  publications 
are  prepared  around  ideas  and  concepts  that  may  include  all 
or  only  a  few  of  a  grant's  original  research  objectives. 

When  researchers  do  not  seek  additional  support — almost  20 
percent  of  those  in  our  review — peer  reviewers  are  not  called 
upon  to  evaluate  the  results  of  the  terminal  grant.  Additionally, 
agency  officials,  who  are  in  the  best  position  to  evaluate  termi¬ 
nal  grants,  do  not  always  read  the  grants'  final  reports  and  pub¬ 
lications  and  compare  them  to  the  grants'  scopes  and  objectives. 

As  a  result,  little  or  no  evaluation  occurs  for  a  significant 
portion  of  the  research  grants  funded  by  NSF  and  NIH. 

NSF  and  NIH  also  perform  general  evaluations  and  analyses  of 
certain  aspects  of  their  grant  processes  or  particular  research 
programs.  For  example,  NSF  is  currently  evaluating  the  publica¬ 
tions  from  a  sample  of  completed  grants  in  its  oceanography  and 
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chemistry  programs  to  determine  if  a  useful  assessment  of  com¬ 
pleted  research  can  be  obtained  by  post-grant  peer  review  of 
publications.  In  this  project,  NSF  is  also  evaluating  possible 
research  quality  indicators. 

NSF  compared  materials  research  results  from  Materials  Re¬ 
search  Laboratories  (MRLs)  it  supports  to  results  of  individually 
funded  research  grants  at  institutions  without  MRLs.  1/  A  compar¬ 
ison  of  sample  publications  resulting  from  each  was  made  to  deter¬ 
mine  if  MRLs  produce  better  materials  research  than  non-MRLs .  The 
study  showed  there  were  some  differences  between  MRL  and  non-MRL 
research  publications.  For  example,  MRL  publications  showed  a 
greater  emphasis  on  experimental  engineering  oriented  research 
and  a  higher  quality  of  procedures  than  those  from  non-MRLs. 

NIH  studied  whether  its  large  grant  programs  produce  results 
consistent  with  their  goals.  The  study  compared  NIH  large  grant 
programs  to  NIH  research  project  grants  to  determine  if  they  are 
comparable  in  quality,  if  projects  are  more  focused  than  programs, 
if  programs  promote  interdisciplinary  research,  and  if  programs 
address  problems  of  high  national  priority.  The  study  concluded 
that  the  programs  do  produce  results  consistent  with  their  goals. 

Publications 


Given  the  apparent  importance  of  publications  in  the  grant 
process,  we  asked  researchers  and  program  officials  if  publica¬ 
tions  are  an  adequate  measure  of  research  success.  They  agreed 
overwhelmingly  that  they  are  a  good  measure  of  research  success. 
Some  cautioned,  however,  that  the  quality  of  the  journal  must 
also  be  considered.  Specifically,  refereed  or  peer  reviewed 
journals  were  cited  as  being  of  the  best  quality.  While  counting 
the  number  of  publications  resulting  from  a  grant  is  the  simplest 
evaluation  technique,  the  prestige  of  the  journal  indicates 
further  the  quality  of  the  results. 

Another  indicator  of  quality  is  use  of  research  results  by 
other  researchers.  Several  researchers  and  program  officials 
mentioned  frequency  of  citation  as  a  measure  of  research  success. 
However,  neither  NSF  or  NIH  use  these  indices  to  aid  in  evaluating 
researchers'  performance.  Other  measures  of  research  success 
mentioned  by  researchers  and  program  officials  included  the  num¬ 
ber  of  graduate  students  trained,  and  standing  in  the  research 
community  as  reflected  by  professional  awards,  presentation  of 
papers  at  conferences,  etc. 


1/Evaluative  Study  of  the  Materials  Research  Laboratory  Program: 
Summary  Report  (Hie  MITRE  Corporation,  McLean,  Virginia, 
September  1978). 
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Negative  results 


Our  review  indicated  that  results  are  often  not  published 
when  researchers  demonstrate  that  a  hypothesis  or  approach  is 
erroneous.  These  results,  known  as  negative  results,  were  pub¬ 
lished  for  11  of  the  26  grants  in  which  they  occurred.  About 
half  the  researchers  interviewed  said  that  negative  results 
should  be  published  and  that  publishing  negative  results  is 
currently  a  common  practice.  Most  of  these  researchers  also 
explained,  however,  that  negative  results  are  disseminated 
informally  throughout  the  research  community  at  scientific  meet¬ 
ings  and  conferences  and  that  they  are  published  only  when  they 
are  significant. 

NSF  and  NIH  requirements  for  reporting  negative  results  vary. 
They  are  not  required  in  NSF  progress  reports,  but  are  in  NlH's. 
Neither  agency  specifically  requires  their  inclusion  in  renewal 
proposal  prograss  reports  or  final  technical  reports. 

COMPARING  SCIENTIFIC  PERFORMANCE 
ACCOUNTABILITY  PROCESSES 

One  of  the  objectives  of  our  review  was  to  compare  the  sci¬ 
entific  performance  accountability  processes,  primarily  peer 
review,  used  by  NSF  and  NIH  to  determine  if  one  process  provided 
better  accountability  than  the  other.  Table  7  compares  these 
processes  for  issues  developed  during  our  review. 

The  comparison  shows  that  for  9  of  the  10  issues  listed,  the 
N:;h  process  provided  better  scientific  performance  accountability 
than  the  processes  used  by  NSF.  The  NSF  panel  process  provided 
better  accountability  than  NIH's  in  comparing  publications  to 
grant  scope  and  objectives.  Both  the  NSF  panel  and  ad  hoc  proc¬ 
esses  were  less  effective  than  NIH  for  five  of  the  issues 
discussed. 
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Table  7 


Comparing  Processes 


Peer  review  method _ 

NIH 

peer  review  NSF  NSF 

Major  issues  group  panel  ad  hoc 

- ( numbers) - 


Renewals  of  sample  grants  awarded 

Other  grants  awarded  by  NSF  or  NIH  not 

19 

14 

13 

related  to  our  sample  grant 

3 

4 

8 

No 

other  grants  awarded  by  NSF  or  NIH 

3 

3 

8 

Sample  grants  reviewed 

25 

21 

29 

1. 

Grants  in  our  sample  where  the  evi¬ 
dence  showed  that  scientific  perfor¬ 
mance  accountability  processes 
were  inadequate 

1 

5 

6 

2. 

Grants  where  peer  review  comments 
impacted  upon  grant  award  decision 

12 

10 

4 

3. 

Evidence  found  that  renewal  peer 
reviewers  evaluated  immediately 
preceding  grant 

95 

29 

15 

4. 

No  evidence  found  that  renewal  peer 
reviewers  evaluated  immediately 
preceding  grant 

5 

71 

85 

5. 

Objectives  of  sample  and  renewal 
grants  the  same 

9 

29 

23 

6. 

Peer  review  comments  sent  to  researcher 

40  a/ 

9 

34 

7. 

Written  summary  of  panel/peer  review 
group  deliberations 

sample  grant 

100 

57 

N/A 

renewal  grant 

100 

64 

N/A 

8. 

Publications  of  immediately  preceding 
grant  identified  in  renewal  proposal 

83 

29 

45 

9. 

Progress  reports  compared  to  grant 
scope  and  objectives 

45 

13 

15 

10. 

Publications  compared  to  grant  scope 
and  objectives 

20 

68 

4 

a/NIH  policy  has  been  changed  to  automatically 

forward 

comments 

to 

researchers . 
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CHAPTER  3 


RESEARCHERS  ACCOUNTABLE  FOR 
RESULTS — NOT  ACCOMPLISHING  OBJECTIVES 

Researchers  varied  significantly  in  accomplishing  their 
grant  objectives.  While  some  researchers  accomplished  all  that 
they  originally  proposed,  most  only  partially  met  their  objec¬ 
tives.  That  most  researchers  did  not  accomplish  all  that  they 
originally  proposed  was  of  little  concern  to  peer  reviewers  and 
program  officials.  The  significance  of  the  researchers'  results 
was  considered  more  important  than  the  degree  to  which  original 
objectives  were  attempted  or  accomplished.  NIH  requires  that 
proposals  clearly  state  the  objectives  to  be  undertaken  during 
the  grant  period,  as  well  as  the  overall  objective(s)  of  the 
research  project,  while  NSF  proposal  requirements  do  not  dis¬ 
tinguish  between  these  types  of  objectives.  Neither  agency 
specifically  spells  out  the  extent  to  which  a  researcher  can 
change  objectives  without  agency  approval. 

We  agree  that  in  determining  whether  a  researcher  should 
receive  additional  funding,  the  results  of  a  research  grant  should 
be  a  more  important  consideration  than  accomplishing  original  ob¬ 
jectives.  However,  based  on  our  review,  we  believe  that  more  spe¬ 
cific  objectives,  guidelines  for  allowing  changes  in  objectives, 
and  the  reporting  of  the  grant  results  in  relation  to  these  objec¬ 
tives  would  assist  peer  reviewers  in  assessing  (1)  the  progress 
made  under  the  preceding  grant,  and  (2)  the  difference  between 
the  preceding  grant  and  the  renewal  proposal.  It  would  also  pro¬ 
vide  NSF  and  NIH  with  more  assurance  that  the  objectives  which 
the  peer  reviewers  thought  had  scientific  merit  were  worked  on. 

ACCOMPLISHING  OBJECTIVES  NOT 
ALWAYS  POSSIBLE 


While  most  researchers  who  were  awarded  renewal  grants 
conducted  research  and  disseminated  their  results,  most  did  not 
accomplish  all  their  grant  objectives.  Forty-six  researchers 
were  awarded  renewal  grants  to  continue  the  same  line  of  research 
as  the  grants  included  in  our  sample.  Only  10  of  these  research¬ 
ers  said  that  all  their  grant  objectives  were  accomplished.  Table 
8  summarizes  the  extent  to  which  objectives  were  met  by  all  46 
researchers: 
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Table  8 


Meeting  Objectives 
NSF 

ad  hoc 


Objectives  all  met  2 


Objectives  partially  met  11 

Objectives  not  met  _0 

Total  r3 

The  reasons  most  often  cited  by  researchers  for  not  fully 
accomplishing  their  objectives  were  budget  cuts  or  the  broad, 
long-term  nature  of  objectives.  Other  less  frequently  mentioned 

reasons  were  changes  in  research  focus  or  new  objectives  being 
developed  during  the  grant  period. 

Only  2  of  the  46  researchers  did  not  disseminate  their  re¬ 
search  results.  They  also  only  partially  accomplished  their 
objectives.  One  of  these  researchers  had  a  1-year  NSF  panel 
grant.  He  had  actively  published  for  several  years  prior  to  this 
grant,  and  peer  reviewers  for  the  proposal  to  renew  this  grant 
were  pleased  with  his  long-term  productivity.  The  second  re¬ 
searcher  had  a  NSF  ad  hoc  grant.  The  researcher  said  that  three 
publications  resulted  from  his  sample  grant.  However,  when  we 
reviewed  these  publications  we  found  that  they  did  not  acknowledge 
the  sample  grant.  Two  of  these  publications  acknowledged  support 
from  a  previous  NSF  grant,  and  the  third  only  acknowledged  NSF 
support  in  general  without  citing  any  specific  grant. 

Three  of  the  46  researchers  did  not  publish  any  of  their 
results  in  peer  reviewed  publications.  They  also  only  partially 
met  their  objectives.  One  of  these  researchers  published  the 
results  of  his  NSF  ad  hoc  grant  in  12  non-peer  reviewed  publica¬ 
tions.  The  second  researcher  discussed  the  slow  progress  made 
during  the  grant  period  in  the  proposal  to  renew  his  NSF  ad  hoc 
grant.  However,  the  proposal  also  indicated  that  his  methods  had 
begun  to  work  at  last.  The  program  official  responsible  for  the 
renewal  said  that  peer  reviewers  indicated  interest  in  continuing 
the  researcher's  study.  The  peer  reviewers  told  us  that  they  had 
seen  the  researcher's  results  at  conferences. 

The  third  researcher  also  presented  the  results  of  his  NSF 
panel  grant  at  scientific  conferences.  We  contacted  the  six  peer 
reviewers  who  reviewed  the  renewal  proposal.  One  advised  us  that 
he  was  aware  of  the  researcher's  results  through  these  conference 
presentations.  Two  reviewers  stated  that  they  read  preprints  of 
publications  and  three  stated  that  the  progress  report  section  of 
the  renewal  proposal  satisfied  them  as  to  the  researcher's  prog¬ 
ress  and  results. 


NSF 

panel  NIH  Total 

3  5  10 

11  14  36 

_0  _0  _0 

14  19  46 
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Although  89  percent  of  the  researchers  did  publish  some  of 
their  results  in  peer  reviewed  journals,  only  50  percent  1_/ 
published  in  time  for  review  by  the  peer  reviewers  of  the  renewal 
proposal.  Thirteen  of  the  19  NIH  grants  which  were  renewed 
resulted  in  publications  which  were  available  at  the  time  the 
renewal  proposal  was  submitted.  Ten  of  the  27  NSF  grants  renewed 
(7  of  13  NSF  ad  hoc,  3  of  14  NSF  panel)  had  publications  available 
at  the  time  of  the  submission  of  the  renewal  proposal.  Without 
publications,  the  peer  reviewers  of  the  renewal  proposal,  in  most 
cases,  must  rely  on  the  progress  report  in  the  renewal  proposal 
to  determine  the  results  of  the  immediately  preceding  grant. 
Therefore,  as  discussed  in  chapter  2,  a  clear,  easily  identified 
presentation  of  preceding  grant  results  and  objectives  would 
provide  peer  reviewers  with  more  adequate  information  upon  which 
to  base  their  evaluation.  Fifty-five  percent  (32  of  58)  of  the 
peer  reviewers  we  contacted  stated  that  having  the  results  of  the 
immediately  preceding  grant  related  to  the  grant's  objectives 
would  be  useful  information  in  evaluating  the  renewal  proposal. 

PROPOSAL  OBJECTIVES  DIFFER 

One  of  the  most  often  cited  reasons  for  not  accomplishing 
objectives  was  the  long-term  nature  of  the  objectives.  NIH  has 
attempted  to  address  this  issue  by  requiring  that  proposals 
include  overall  research  objectives  or  long-term  goals,  which  it 
calls  "objectives,"  as  well  as  specific  objectives  for  the  period 
of  requested  support,  which  it  calls  "specific  aims."  NSF  does 
not  specify  whether  the  "objectives"  it  requires  researchers  to 
include  in  proposals  are  long-  or  short-term. 

Comparing  the  objectives  for  the  grant  in  our  review  with 
renewal  grant  objectives  showed  that  objectives  generally  were 
not  the  same.  The  highly  technical  nature  of  these  objectives 
made  such  comparisons  difficult.  However,  nine  sample  grant 
researchers  appeared  to  have  some  of  the  same  objectives  for 
their  sample  and  renewal  grants.  Work  plans,  however,  changed 
between  the  two  grants,  suggesting  that  these  objectives  were 
long-term  goals.  NIH's  policy  of  having  specific  aims  for  the 
requested  grant  period  appears  to  have  made  researchers  more 
definitive  in  their  objectives  for  specific  grant  periods.  The 
proportion  of  NIH  researchers  having  the  same  specific  aims  for 
both  grants  was  considerably  less  than  the  proportion  of  NSF 
researchers  having  the  same  objectives  for  both  grants.  Only  2 
of  the  19  NIH  researchers  having  renewal  grants  had  some  of  the 
same  specific  aims.  In  contrast,  7  of  the  27  NSF  researchers 
having  renewal  grants  had  some  of  the  same  objectives. 


1_/The  publications  either  solely  or  partially  acknowledged  the 
grant  we  reviewed. 
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NIH  also  requires  that  renewal  proposals  identify  the  speci¬ 
fic  aims  from  the  preceding  grant  and  an  account  of  the  published 
and  unpublished  results  indicating  progress  toward  the  achievement 
of  the  specific  aims.  NSF  simply  requires  a  summary  of  progress 
to  date  and  its  relation  to  the  proposed  work.  Two  NSF  ad  hoc 
grants  demonstrate  the  problems  this  vague  proposal  requirement 
can  cause  peer  reviewers  in  evaluating  a  renewal  proposal  (see 
page  23).  In  one  instance,  a  peer  reviewer  was  able  to  summarize 
his  problem  very  clearly.  In  the  second  instance,  a  peer  reviewer 
had  problems  evaluating  the  renewal  proposal  because  the  proposal 
did  not  clearly  detail  the  results  of  the  preceding  grant.  The 
accomplishments  were  somewhat  different  from  what  was  originally 
proposed.  Although  the  researcher’s  work  was  within  the  subject 
area  of  the  grant,  he  added  some  objectives  as  the  research  pro¬ 
gressed.  Without  a  clear  statement  of  the  objectives  and  results 
of  the  preceding  grant,  the  peer  reviewers  would  generally  not 
know  about  the  added  work. 

At  NSF,  as  indicated  in  chapter  2,  a  clear  statement  of  the 
objectives  and  results  of  the  preceding  grant  in  the  renewal 
proposal  is  important  because,  typically,  different  peer  reviewers 
evaluate  the  renewal  proposal  than  evaluated  the  preceding  grant 
proposal.  Twenty-seven  NSF  grants  in  our  review  were  renewed. 

In  about  37  percent  of  these  cases,  none  of  the  same  peer  review¬ 
ers  were  used  to  evaluate  the  preceding  grant  and  the  renewal 
and  in  about  52  percent  of  the  cases  only  one  peer  reviewer  was 
the  same. 

For  the  seven  renewed  grants  discussed  earlier  in  which 
some  of  the  objectives  for  the  preceding  grant  and  the  renewal 
were  the  same,  different  peer  reviewers  were  used  in  three  cases, 
one  peer  reviewer  repeated  in  three  cases  and  three  repeated  in 
one  case. 

RESEARCH  IS  RESULTS -ORIENTED 


According  to  researchers,  peer  reviewers,  and  program  offi¬ 
cials,  researchers  are  primarily  evaluated  on  the  significance  of 
their  research  and  results.  Hence,  the  degree  to  which  research 
objectives  are  accomplished  is  of  lesser  importance.  Research  is 
results-oriented  because  of  (1)  the  creative  nature  of  basic  re¬ 
search;  (2)  NSF  and  NIH  grant  policies  allowing  researchers  to 
deviate  from  objectives;  and  (3)  the  speculative  nature  of 
objectives . 

Allowing  for  creativity 

While  research  objectives  are  important  in  making  grant  award 
decisions,  they  become  less  important  during  the  grant  period. 

The  creative  nature  of  basic  research  lessens  their  importance  be¬ 
cause  the  research  process  and  results  may  differ  from  what  was 
initially  envisioned.  Researchers  and  program  officials  charac¬ 
terize  basic  research  as  exploratory;  results  are  unpredictable. 
Unforeseen  problems  make  research  risky.  Spinoff  ideas  having 
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more  significance  than  those  initially  proposed  may  emerge  during 
a  grant  period  and  should  be  pursued.  As  one  researcher  said, 
research  proposals  are  merely  a  point  of  departure. 

Allowing  for  change 

Change  in  research  direction  during  a  grant  is  recognized  as 
a  legitimate  part  of  the  research  process.  Both  NSF  and  NIH 
grant  policies  allow  such  changes,  and  several  researchers  did 
deviate  from  their  initial  objectives.  NSF  and  NIH  instructions, 
however,  do  not  specify  how  much  leeway  researchers  have  in 
changing  their  objectives  before  they  must  obtain  agency  approval 
for  doing  so.  NSF  instructions  require  prior  approval  for  any 
changes  in  objectives  and  for  "significant"  changes  in  methods  or 
procedures.  NIH  instructions  state  only  that  researchers  are 
required  to  obtain  prior  approval  for  "major"  changes  in  scope. 

Most  researchers  said  they  have  or  should  have  leeway  to 
change  objectives.  Most  also  said  ♦‘hat  they  would  notify  the 
agency  if  they  changed  f ields/s ub ject  areas  or  objectives  as 
opposed  to  the  scope  of  work  within  these  objectives.  None  of 
the  researchers  included  in  our  review  said  that  they  had  materi¬ 
ally  changed  the  subject  matter  of  their  research  from  what  they 
had  initially  proposed.  Although  12  researchers  did  pursue  some 
objectives  not  in  their  sample  grant  proposals,  they  stated  that 
all  these  new  objectives  were  in  the  same  subject  area.  In  these 
instances,  the  research  objectives  presented  in  the  proposals 
were  also  frequently  only  partially  completed. 

Other  researchers  did  not  attempt  all  the  research  objectives 
presented  in  their  proposals.  Program  officers  (19  of  22)  gener¬ 
ally  said  that  not  attempting  all  the  objectives  presented  in  a 
proposal  is  not  a  change  in  research  direction.  Three  others 
added  that  not  attempting  the  major  objective  or  central  theme  of 
a  proposal  would  constitute  a  change  in  research  direction. 

Researchers  who  deviated  from  their  initial  objectives  varied 
in  their  views  on  seeking  agency  approval  for  these  changes.  For 
example,  one  researcher  only  accomplished  one  of  five  initially 
proposed  research  objectives.  The  researcher  said  that  during 
the  course  of  the  grant  it  became  apparent  that  unanticipated 
research  had  to  be  performed  before  the  initially  proposed  objec¬ 
tives  could  be  attempted.  The  researcher  contacted  NSF  regarding 
this  problem  prior  to  conducting  the  unanticipated  research. 

A  second  researcher  thought  researchers  should  have  the 
freedom  to  change  direction  without  contacting  the  agency  regard¬ 
ing  the  change.  He  was  aware  of  NSF's  policy  requiring  contact 
before  changing  research  direction,  but  he  felt  researchers  did 
not  adhere  to  this  policy.  He  added  that  a  significant  change  in 
research  direction  would  have  to  occur  to  warrant  agency  contact. 

A  third  researcher  felt  that  researchers  who  adopt  a  new 
direction  in  their  research  should  be  allowed  some  time  after 
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making  the  change  before  they  must  notify  the  agency  of  the 
change.  He  explained  that  this  delay  would  allow  researchers  to 
use  the  time  following  their  adption  of  a  new  direction  to  assure 
themselves  that  the  new  direction  was  viable.  He  said  it  would 
then  be  appropriate  to  notify  the  agency  in  the  renewal  proposal. 

A  fourth  researcher  said  that  he  did  not  fully  complete  his 
broadly  stated,  long-term  objective.  He  explained  that  during  the 
research  he  developed  a  new  technique  which  changed  his  priorities 
from  the  proposed  scope  of  work  to  other  new  experiments.  He 
said  he  notified  NSF  of  this  change  via  his  final  report. 

Several  researchers  and  program  officials  commented  that  it 
is  a  resercher ' s  prerogative  to  change  research  direction  during 
the  course  of  a  grant.  They  added,  however,  that  if  such  changes 
do  not  produce  significant  results,  adverse  peer  evaluations  will 
occur  when  renewal  proposals  are  submitted  for  continued  support. 

We  would  agree  that  the  nature  of  basic  research  requires 
that  researchers  have  the  freedom  to  change  their  research  as  it 
progresses.  However,  reporting  results  as  they  relate  to  the 
originally  proposed  objectives  would  not  stifle  any  need  to 
change.  Results  are  and  still  would  be  the  most  important  con¬ 
sideration  in  deciding  if  the  researcher  should  be  funded  again. 
This  reporting  requirement  would  provide  better  accountability 
because  it  becomes  easier  for  peer  reviewers  to  judge  the  progress 
made  under  the  grant  if  they  know  what  the  researcher  was  striving 
to  do.  Secondly,  it  is  easier  for  peer  reviewers  to  judge  the 
difference  between  the  preceding  grant  and  the  renewal  proposal. 
Finally,  it  offers  the  funding  agency  some  assurance  that  the 
objectives  which  the  peer  reviewers  deemed  to  have  scientific 
merit  were  worked  on. 

A  more  specific  definition  is  needed,  however,  as  to  what 
constitutes  a  change  requiring  prior  agency  approval.  The  four 
researchers  discussed  above  each  interpreted  NSF  and  NIH  guide¬ 
lines  differently  as  to  when  to  let  the  agency  know  about  a 
change.  One  didn't  tell  the  agency,  one  contacted  the  agency 
prior  to  the  change,  and  two  told  the  agency  after  the  fact — one 
in  a  final  report  and  one  in  a  renewal  proposal.  A  "significant" 
or  "major"  change,  as  the  NSF  and  NIH  criteria  respectively  now 
state,  does  not  appear  specific  enough  to  avoid  this  confusion 
and  diversity.  Researchers  could  unknowingly  change  their 
research  to  do  something  already  being  worked  on  by  someone  else. 
More  specific  guidelines  as  to  when  agency  approval  is  required 
would  alleviate  this  problem.  Also,  a  reporting  of  objectives 
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in  the  renewal  proposal  would  at  least  alert  the  agency  that  the 
change  was  made.  1/ 

Research  objectives  are  speculative 

Most  of  the  researchers,  peer  reviewers,  and  program  offi¬ 
cials  we  contacted  did  not  expect  all  grant  objectives  to  be 
accomplished.  Objectives  were  generally  characterized  as  having 
broad  long-term  changing  goals.  Uncertainty  surrounds  objectives 
because  researchers  cannot  predict  what  will  happen  once  research 
is  started,  research  may  continue  into  subsequent  grants,  and 
relating  objectives  to  published  results  can  be  difficult. 

In  contrast  to  NSF,  NIH  grant  procedures  appear  to  reduce 
the  uncertainty  surrounding  objectives  by  requiring  researchers 
to  distinguish  between  long-term  objectives  and  specific  aims. 

That  NIH  requires  researchers  to  report  on  objectives  does  not 
necessarily  mean  that  researchers  are  held  accountable  for  accom¬ 
plishing  these  objectives.  NIH  requires  researchers  to  report  the 
progress  made  toward  meeting  the  grant  objectives  which  the  peer 
reviewers  deemed  to  have  scientific  merit.  Of  the  19  NIH  re¬ 
searchers  awarded  renewal  grants,  14  said  that  they  only  partially 
met  their  grant  objectives,  suggesting  that  the  grant  results 
were  significant  enough  to  warrant  continuing  support  even  though 
not  all  objectives  were  met. 

Table  9  shows  that  the  majority  of  researchers  and  program 
officials  did  not  expect  grant  objectives  to  be  accomplished. 

However,  many  of  the  peer  reviewers  who  expected  a  grant's 
objectives  to  be  accomplished  qualified  their  expectation  by 
stating  that  results  are  still  the  most  important  consideration 
when  they  review  proposals.  Although  these  individuals  did  not 
expect  the  objectives  to  be  accomplished,  many  of  them  did  state 
that  they  expect  researchers  to  try.  It  is  easier  to  assess  if  a 
researcher  attempted  the  objectives  if  the  objectives  of  the 
preceding  grant  are  identified  in  the  renewal  proposal. 


1/Any  change  that  amounts  to  a  change  in  the  scope  of  the  grant 
will  cancel  the  existing  grant  and  have  the  effect  of  deobligat- 
ing  that  grant.  Unless  the  changed  grant  is  approved  either 
formally  or  informally,  the  costs  associated  with  it  cannot  be 
allowed.  If  the  appropriation  from  which  the  original  grant 
was  made  is  no  longer  available,  the  costs,  where  allowable, 
must  be  assigned  to  an  appropriation  currently  available.  See 
57  Comp.  Gen.  459  (1978). 
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Table  9 


Expecting  Objectives 

to  be  Accomplished 

NSF 

NSF 

Researchers 

(Ad  Hoc) 

( Panel ) 

NIH 

Total 

Yes 

10 

6 

6 

22 

No 

13 

11 

16 

40 

No  response 

_6 

4 

3 

13 

. . . 

29 

21 

25 

75 

— 

Program  officials 

Yes 

2 

2 

2 

6 

No 

15 

9 

15 

39 

Sometimes 

0 

0 

0 

0 

No  response 

_0 

_0 

_2 

3 

17 

11 

20 

48 

— 

Peer  reviewers 

Yes 

11 

12 

4 

27 

NO 

13 

5 

7 

25 

No  response 

4 

1 

1 

6 

' 

28 

18 

12 

58 

— 

Objectives  were 

generally  characterized  as  speculative  long 

term  changeable  ideas 

.  Typical  comments  on 

the  nature  of 

objectives  by  researchers  were: 

--Researchers  should  have  a 

lot  of  leeway  to 

change  their 

objectives  and 

scope  since 

research 

tends  to  branch  out 

into  different  directions  as  discoveries  are  made  and  new 
scientific  ideas  arise. 

— Objectives  are  general  goals  toward  which  a  researcher 
moves.  In  pure  research,  it  is  impossible  to  have  firm 
objectives  similar  to  those  in  applied  research.  [He  did 
not  expect]  objectives  to  be  accomplished  because  results 
cannot  be  guaranteed  in  basic  research. 

--Objectives  outline  what  a  researcher  hopes  to  accomplish 
during  a  grant  period,  and  in  another  sense,  they  are  the 
researcher's  lifetime  goals.  [The  researcher  felt]  objec¬ 
tives  should  be  stated  more  broadly  than  they  currently 
are . 

— Grants  are  a  continuum.  Hence,  while  some  research  may  be 
completed,  other  research  will  only  "progress"  and  will 
continue  in  subsequent  grants. 
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Typical  comments  by  peer  reviewers  included: 

--Any  expectations  about  objectives  must  be  viewed  in  the 
context  of  the  researcher  proposing  them.  Some  researchers 
tend  to  be  optimistic  and  broad  in  proposing  research, 
while  others  are  very  explicit.  At  the  very  least  some 
progress  [would  be  expected]  in  the  direction  of  proposed 
objectives  during  a  grant  period. 

--Expecting  objectives  to  be  accomplished  is  very  naive 
because  a  researcher  cannot  predict  what  will  occur  once 
research  is  started. 

A  comment  by  a  program  officer: 

— It  is  unrealistic  to  expect  objectives  to  be  accomplished 
in  one  grant  period  for  a  theory-type  grant.  Many  re¬ 
searchers  obtain  renewal  grants  to  continue  working  toward 
the  same  objectives.  In  contrast,  a  peer  reviewer  said 
that  in  experimental  grants  some  experiments  have  a  high 
probability  of  success. 

Objectives  may  be  difficult  to  relate  to  published  results. 
Forty-six  percent  of  the  program  officers  interviewed  said  that 
they  do  not  examine  publications  to  determine  which  objectives 
researchers  have  met.  One  reason  is  that  publications  stress 
results  and  do  not  necessarily  describe  objectives.  Even  when 
publications  do  describe  objectives,  program  officials  said 
publications  are  not  used  for  this  purpose  because  doing  so  is 
too  time-consuming  and  difficult.  Many  program  officials  said 
they  rely  instead  on  explicit  written  acknowledgements  in  publi¬ 
cations  to  identify  which  grant  supported  a  publication  when 
they  are  otherwise  in  doubt.  Reading  publications  may  not  always 
alleviate  the  program  officer's  doubt,  however.  Our  analysis  of 
publications  resulting  from  the  grants  reviewed  indicated  that, 
in  many  cases,  the  publication  does  not  identify  the  grant  which 
supported  the  work. 
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CHAPTER  4 


CONCLUSIONS,  RECOMMENDATIONS,  AND 
MATTERS  FOR  CONSIDERATION  BY  THE  CONGRESS 


CONCLUSIONS 


The  individual  researcher  is  the  keystone  of  the  basic 
research  process.  For  this  reason,  the  extent  of  the  scientific 
performance  accountability  provided  by  the  peer  review  process 
and  the  funding  agency  is  somewhat  limited.  Additionally,  there 
is  no  mechanism,  nor  probably  can  there  be  for  individuals  outside 
of  the  research  community  to  provide  this  accountability.  Even 
the  universities,  which  theoretically  are  in  a  position  to  provide 
it,  practically  cannot  and  do  not  because  of  the  "academic  free¬ 
dom"  environment  and  the  lack  of  expertise  about  the  research 
being  performed.  Therefore,  given  these  constraints,  NSF  and  NIH 
research  grants  should  be  subjected  to  a  thorough  scientific 
performance  accountability  review.  Also,  the  scientific  account¬ 
ability  process  should  not  be  concerned  only  with  the  process 
alone.  The  process  is  merely  a  "means  to  an  end  and  not  the  end 
itself."  It  exists  to  serve  a  purpose--that  of  enabling  the  best 
evaluation  possible  of  research  proposals  to  determine  «-hose  that 
most  merit  funding. 

Although  the  peer  review  and  internal  review  systems  NSF  and 
NIH  use  to  provide  scientific  accountability  over  basic  research 
grants  are  working  reasonably  well,  improvements— especially  at 
NSF — are  needed.  Of  the  two  agencies'  systems,  the  NIH  system 
contains  better  safeguards  and  therefore  provides  more  assurance 
that  the  research  supported  is  attempted  and  that  unproductive 
researchers  will  be  identified  and  prevented  from  receiving  con¬ 
tinued  funding.  This  conclusion  is  based  primarily  on  the  fact 
that  the  NIH  system  provides  (1)  more  specific  instructions  to 
peer  reviewers  for  judging  the  quality  of  research  performed  and 
proposed,  and  (2)  better  identification  in  proposals  of  research 
results  and  their  relation  to  research  objectives  of  immediately 
preceding  grants.  Additionally,  the  NIH  system  makes  more  effec¬ 
tive  use  of  peer  reviewer  comments  and  better  documents  its  panel 
peer  review  deliberations. 

At  NSF,  more  specific  guidance  to  peer  reviewers  (the  first 
point  above)  is  needed  in  part,  because  in  many  instances  program 
officers  do  not  have  the  expertise  to  make  judgments  about  the 
research  proposed  and  performed.  Better  identification  (the 
second  point  above)  is  needed  because  in  many  instances  (1)  the 
proposal  provides  the  only  source  of  a  grant's  results  at  the  time 
of  renewal,  and  (2)  different  program  officers  and  peer  reviewers 
evaluate  the  renewal  proposal  than  evaluated  the  preceding  propo¬ 
sal.  This  information  will  enable  peer  reviewers  and  program 
officers  to  determine  how  successful  the  researcher  was  in  achiev¬ 
ing  the  preceding  grant's  objectives,  if  the  objectives  changed, 
the  difference  between  the  proposed  work  and  work  accomplished. 
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and  any  publications  which  resulted  from  the  immediately  preceding 
grant . 

There  are,  however,  problems  common  to  both  agencies.  New 
project  proposals  do  not  include  information  on  the  immediately 
preceding  but  unrelated  grant  and  peer  reviewers  are  not  provided 
the  final  technical  report  or  listing  of  publications  resulting 
from  the  prior  grant.  Additionally,  the  monitoring  of  ongoing 
NSF  and  NIH  grants  by  program  officials  is  not  systematic. 

At  both  NSF  and  NIH,  many  research  projects  are  funded  by  a 
series  of  successive  grants,  each  grant  having  a  duration  of  from 
1  to  5  years.  When  a  researcher  seeks  to  renew  a  grant,  we  be¬ 
lieve  NSF,  like  NIH,  should  require  more  specific  information 
about,  and  require  peer  reviewers  to  evaluate  and  comment  on  the 
progress  and  accomplishments  of,  the  immediately  preceding  grant 
as  one  of  the  factors  in  deciding  whether  to  fund  the  researcher 
again . 

NSF  does  not  ask  peer  reviewers  to  specifically  evaluate 
researchers'  performance  on  immediately  preceding  grants  when 
researchers  seek  renewal  grants.  The  reviewers  are  only  asked 
to  evaluate  the  scientific  merit  of  the  proposal  and  the  re¬ 
searcher's  track  record--including  recent  accomplishments.  Our 
analysis  of  peer  review  comments  for  researchers  seeking  renewal 
grants  from  NSF  showed  that  in  only  22  percent  of  these  cases 
were  evaluative  comments  made  by  the  peer  reviewers  regarding 
the  researchers'  immediately  preceding  grant. 

In  contrast,  at  NIH,  where  peer  reviewers  are  asked  to  com¬ 
ment  on  the  immediately  preceding  grant,  we  found  that  in  95 
percent  of  the  cases  the  peer  reviewers  did  comment  on  the  re¬ 
searcher's  performance  on  the  immediately  preceding  grant. 

Even  though  NSF  requests  peer  reviewers  to  evaluate  a  re¬ 
searcher's  recent  accomplishments,  this  may  not,  in  many  cases, 
be  adequate.  Two-thirds  of  the  researchers  included  in  our 
review  had  more  than  one  grant.  For  these  researchers,  when  peer 
reviewers  evaluate  a  renewal  proposal,  the  researcher's  recent 
accomplishments  might  result  from  a  grant  other  than  the  one 
being  renewed. 

Further,  if  peer  reviewers  were  asked  to  comment  on  the  imme¬ 
diately  preceding  grant,  the  inadequacy  of  current  NSF  proposal 
requirements  would  make  such  an  evaluation  difficult.  Researchers 
in  their  renewal  proposal  progress  reports  are  not  required  to 
cite  the  objectives  of  the  immediately  preceding  grant  nor  iden¬ 
tify  the  resulting  publications  or  other  output.  Without  the 
objectives,  it  becomes  a  more  difficult  task  for  peer  reviewers 
to  determine  how  the  proposed  work  differs  from  the  work  under 
the  preceding  grant.  Without  identifying  the  publications  from 
the  preceding  grant,  it  is  more  difficult  for  peer  reviewers  to 
evaluate,  for  multi-funded  researchers,  specific  grant  performance. 
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NIH  also  makes  more  effective  use  of  peer  review  comments 
by  automatically  releasing  them  to  researchers.  NSF's  policy  is 
to  release  them  only  when  the  researcher  requests  them.  NSF's 
policy  of  not  routinely  providing  this  information  to  all  re¬ 
searchers  has  adversely  affected  the  conduct  of  the  research  in 
some  instances.  Further,  for  many  NSF  grants  which  are  panel 
reviewed,  no  summary  of  the  panel  deliberations  exists  to  be 
released  to  the  researcher. 

NSF  and  NIH  also  have  problems  which  are  common  to  their 
accountability  processes.  When  multi-funded  researchers  submit 
new  project  proposals,  neither  agency  requires  the  researchers 
to  discuss  the  prior  grant  or  identify  its  publications  in  the 
new  proposal.  Also,  peer  reviewers  are  not  specifically  asked 
to  evaluate  the  prior  grant's  results,  other  than  as  part  of  the 
researcher's  general  track  record,  and  they  are  not  provided  with 
the  final  technical  report  or  listing  of  publications  resulting 
from  the  prior  grants.  These  procedures  do  not  preclude  re¬ 
searchers  from  avoiding  accountability  on  unproductive  grants  by 
seeking  grants  for  new  projects. 

NSF  and  NIH  do  not  monitor  scientific  progress  during  grant 
periods  in  a  uniform  manner.  Progress  reports  submitted  by 
researchers  are  the  primary  tool  available  for  performing  this 
function.  Yet,  program  officials  at  both  agencies  use  them  in  a 
variety  of  ways — some  just  read  them,  some  compare  them  to  the 
grant's  original  scope  and  objectives  and,  at  NIH,  some  officials 
prepare  a  checklist  as  evidence  of  progress  report  review.  A 
more  systematic  and  uniform  approach  at  both  agencies  would 
provide  more  effective  scientific  performance  accountability. 

Most  of  the  researchers  who  were  awarded  renewal  grants  did 
not  accomplish  all  of  the  objectives  of  the  immediately  preceding 
grant.  Peer  reviewers  and  program  officers  were  not  concerned  by 
this  fact  because  they  believe  that  the  results  of  the  grant  are 
more  important  than  the  accomplishment  of  originally  proposed 
objectives  and  because,  for  the  most  part,  they  do  not  expect  all 
grant  objectives  to  be  accomplished.  Most  of  the  peer  reviewers 
and  program  officers,  however,  did  expect  the  researchers  to 
attempt  the  grant's  objectives.  It  is  difficult  to  determine  if 
a  researcher  attempted  the  grant's  objectives,  which  the  peer 
reviewers  thought  had  scientific  merit,  if  the  renewal  proposal 
does  not  identify  what  the  preceding  grant’s  objectives  were. 

NSF,  unlike  NIH,  does  not  require  renewal  proposals  to  restate 
the  preceding  grant's  objectives. 

The  majority  of  peer  reviewers  we  contacted  believed  that 
having  the  accomplishments  of  the  preceding  grant  related  to  the 
objectives  would  be  useful  in  evaluating  the  renewal  proposal. 

Our  review  provided  examples  of  instances  where  peer  reviewers 
for  NSF  grants  had  problems  determining  the  progress  of  the 
preceding  grant  and  the  difference  between  the  preceding  grant 
and  the  renewal  proposal.  The  reporting  of  objectives  in  NSF 
renewal  proposals  is  of  even  more  importance  because,  typically. 
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different  peer  reviewers  evaluate  the  renewal  proposal  than 
evaluated  the  proposal  from  the  preceding  grant. 

The  nature  of  long-term  grant  objectives  was  one  of  the  most 
cited  reasons  by  researchers  for  not  accomplishing  grant  objectives. 
This  continuation  of  research  became  apparent  from  our  comparison 
of  objectives  of  the  prior  grant  in  our  review  with  the  objectives 
of  the  renewal  grant.  This  comparison  showed  that  at  NSF  the  two 
grants  had  some  of  the  same  objectives  more  than  three  times  as 
often  as  NIH  grants.  This  is  attributable  to  the  fact  that  NIH 
requires  researchers  to  specifically  state  the  objectives  (spe¬ 
cific  aims)  to  be  attempted  during  the  grant  period  as  well  as 
the  overall  objective  of  the  line  of  research.  NSF  does  not  dis¬ 
tinguish  between  these  types  of  objectives.  NSF  peer  reviewers 
and  program  officers  would  get  a  more  accurate  picture  of  what 
the  researcher  intends  to  do  during  the  grant  period  if  NSF 
required  specific  grant  period  objectives. 

The  extent  to  which  researchers  can  deviate  from  a  grant's 
original  objectives  without  prior  agency  approval  is  left  pri¬ 
marily  up  to  the  researcher.  While  most  researchers  said  that 
they  would  notify  the  agency  beforehand  if  they  were  going  to 
change  research  areas,  a  few  said  they  wouldn't,  while  others 
generally  said  that  they  have  a  lot  of  leeway  in  this  regard. 

Since  the  peer  reviewers  had  previously  determined  that  the  objec¬ 
tives  in  the  grant  proposal  had  scientific  merit,  changing  the 
objectives  tends  to  complicate  the  peers'  scientific  merit  deter¬ 
mination.  Under  these  conditions,  researchers  could  deviate  from 
approved  objectives  by  attempting  objectives  which  were  not 
reviewed  by  peer  reviewers  for  scientific  merit.  More  specific 
guidelines  are  needed  as  to  the  extent  research  objectives  can 
be  changed  without  agency  approval. 

RECOMMENDATIONS 

To  provide  for  more  effective  scientific  performance  account¬ 
ability,  we  recommend  that  the  Director  of  NSF  require  that: 

— Renewal  proposal  progress  reports  identify  the  objectives, 
evidence  of  progress  toward  their  achievement,  any  major 
changes  in  direction  or  emphasis  and  rationale  for  such 
changes,  publications,  and/or  other  output  from  a  re¬ 
searcher's  immediately  preceding  grant. 

— Peer  reviewers  be  asked  when  reviewing  renewal  proposals 
to  specifically  comment  on  a  researcher's  performance 
on  the  immediately  preceding  grant. 

--The  documentation  of  panel  peer  review  deliberations  in¬ 
clude  the  major  elements  required  of  the  NIH  peer  review 
group  summary  statement  when  individual  peer  reviewers' 
written  reviews  do  not  provide  this  information. 
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--Peer  review  comments  be  automatically  sent  to  researchers. 

— Proposals  identify  the  research  objectives  to  be  under¬ 
taken  during  the  grant  period. 

We  recommend  that  the  Directors  of  NSF  and  NIH  require  that: 

— Proposals  for  new  projects  include  evidence  of  progress 
from  the  prior  grant (s). 

--Peer  reviewers  be  furnished  any  available  final  technical 
reports  and  listings  of  publications  from  the  prior  grant (s) 
when  researchers  seek  funding  for  new  projects. 

--More  systematic  and  uniform  review  of  annual  progress  re¬ 
ports  be  made  by  the  program  officers. 

--More  specific  guidelines  be  established  regarding  the  ex¬ 
tent  to  which  researchers  can  change  grant  objectives 
without  prior  agency  approval. 

AGENCY  COMMENTS 
AND  OUR  RESPONSE 

The  National  Science  Foundation  (NSF),  and  the  Department 
of  Health  and  Human  Services  (HHS)  reviewed  and  commented  on  a 
draft  of  this  report.  HHS  stated  the  report  fairly  presents  the 
issues  involved,  and  agreed  with  all  of  our  recommendations  to 
the  Director  of  NIH.  NSF  noted  that  while  improvements  should 
always  be  sought  in  any  system,  any  changes  must  be  considered 
in  the  context  of  workload  implications.  Both  agencies  generally 
concurred  with  our  recommendations  and,  with  one  exception  by 
NSF,  agreed  to  examine  current  practices  and/or  develop  better 
guidelines  to  implement  them.  We  asked  six  universities  to 
comment  on  draft  report  excerpts  dealing  with  university  proposal 
submission  and  monitoring  of  research.  We  received  comments 
from  two  universities  in  time  to  be  incorporated  in  this  report 
(see  appendixes  II-V  with  our  responses  to  their  comments  on 
specific  points) . 

NSF  agreed  with  all  the  recommendations  made  to  it  with  one 
exception.  However,  the  actions  NSF  agreed  to  take  are  based 
somewhat  on  a  misinterpretation  of  the  intent  of  several  of  the 
recommendations — a  misinterpretation  that  needs  to  be  clarified. 

Our  recommendations  are  aimed  at  assuring  that  adequate  informa¬ 
tion  is  contained  in  research  proposals  regarding  the  objectives, 
any  changes  in  direction,  progress,  and  output  under  the  immedi¬ 
ately  preceding  grant.  Our  recommendations  are  also  aimed  at 
all  proposals  whether  from  researchers  with  multiple  grants  or 
only  one  grant  and  whether  for  continued  support  of  an  existing 
project  or  for  support  of  a  new  project  if  the  researcher  already 
had  a  grant. 
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NSF  interpreted  the  recommendations  as  meaning  that  only  a 
greater  distinction  be  made  between  long-  versus  short-term  objec¬ 
tives  in  proposals  from  only  those  researchers  having  multiple 
grants.  This,  however,  is  not  the  intent  of  our  recommendations. 
The  problems  discussed  in  chapter  2,  while  occurring  more  fre¬ 
quently  when  researchers  have  multiple  grants,  also  occur  when 
researchers  have  only  one  grant.  More  important,  however,  more 
information  is  needed  in  research  proposals  to  resolve  the  prob¬ 
lems  in  scientific  performance  accountability,  identified  in 
this  report,  than  just  a  greater  distinction  between  long-  versus 
short-term  objectives.  More  information  is  needed  in  the  progress 
report  sections  of  all  proposals  regarding  the  objectives,  changes 
in  objectives,  progress  toward  the  objectives  and  output  under 
the  immediately  preceeding  grant  from  all  researchers. 

The  suggestion  in  our  draft  report  that  NSF  took  exception 
to  has  been  modified  to  reflect  NSF's  views  and  our  concern  that 
adequate  information  be  available  to  show  the  peer  review  that 
was  done.  The  suggestion  concerned  documentation  of  NSF's  panel 
peer  review  summaries  along  the  lines  of  NIH's.  NSF  said  that 
individual  written  peer  reviewers'  reviews  mitigated  the  need 
for  requiring  a  detailed  summary  such  as  NIH  requires  since  NIH 
usually  does  not  have  individual  written  reviews  and  NSF  does. 

We  agree  with  NSF  and  accordingly  are  recommending  that  NSF's 
panel  summaries  should  show  the  kind  of  information  required  in 
NIH  panel  summaries  only  when  such  data  do  not  appear  in  any  of 
the  individual  written  reviews. 

NSF's  main  concern  about  implementing  our  recommendations 
regards  the  additional  staff  time  that  might  be  required.  Imple¬ 
menting  the  recommendations,  with  the  exception  of  the  one  dealing 
with  panel  summaries,  will  require  no  additional  staff  time.  In 
fact  implementing  the  recommendations  will  probably  save  staff 
time  in  that  proposal  evaluation  will  be  easier  and  thus  less 
time-consuming.  • 

NSF  noted  that  our  sample  resulted  in  an  abnormally  large 
fraction  of  cases  of  researchers  with  multiple  grants  because  only 
grants  to  a  few  large,  prestigious  universities  were  selected. 

Our  sample  of  grants  was  drawn  from  major  research  universities 
in  part  because  they  get  the  bulk  of  NSF  funds.  According  to  NSF 
statistics,  the  top  20  received  40  percent  of  all  NSF  funds  in 
fiscal  year  1980.  The  top  50  got  81  percent  and  the  top  100  got 
78  percent.  It  necessarily  follows  that  wherever  the  greatest 
concentration  of  researchers,  research  grants  and  grant  funds, 
is  where  there  is  the  greatest  need  to  ensure  scientific  perform¬ 
ance  accountability.  Researchers  with  multiple  grants  are  concen¬ 
trated  in  major  research  institutions.  Since  the  major  problem 
is  with  researchers  with  multiple  grants,  we  believe  the  sample 
appropriately  represents  the  bulk  of  NSF  grant  funds  and  points 
up  problems  that  are  net  limited  to  a  small  fraction  of  NSF 
grants . 
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MATTERS  FOR  CONSIDERATION 
BY  THE  CONGRESS 


Accountability  for  public  funds  spent  on  basic  research 
grants  to  colleges  and  universities  continues  to  present  a  para¬ 
dox.  Universities  and  university  researchers  want  more  freedom 
and  less  Government  control  over  how  actual  dollars  are  spent. 

The  Congress  and  the  taxpayers  want  assurance  that  tax  monies 
are  invested  wisely.  But,  the  methods  by  which  that  assurance 
is  provided  should  carefully  avoid  detailed  controls  over  re¬ 
searchers'  activities  that  do  not  siginif icantly  improve  account¬ 
ability,  and  which  can  also  suppress  the  freedom  of  inquiry  so 
essential  to  basic  research.  Our  recommendations  are  intended 
to  improve  scientific  performance  accountability  and  peer  review 
without  inhibiting  researchers'  freedom  of  inquiry. 

The  House  Committee  on  Government  Operations  and  the  Senate 
Committee  on  Governmental  Affairs  during  oversight  hearings  on 
NSF,  NIH,  or  on  matters  relating  to  universities  and  research 
grant  accountability,  should  consider  the  effectiveness  of  the 
scientific  performance  accountability  systems  at  NIH  and  NSF. 

Also,  the  House  Committees  on  Science  and  Technology  and  Appropri¬ 
ations  and  the  Subcommittee  on  Appropriations — HUD  and  Independent 
Agencies,  Senate  Committee  on  Appropriations,  during  their  fiscal 
year  1983  budget  hearings,  should  consider  NSF  and  NIH  actions 
taken  to  improve  the  research  grant  scientific  performance  ac¬ 
countability  systems  since  the  hearings  mentioned  on  pages  6 
and  7.  These  systems  determine  the  quality  of  much  of  the  basic 
research  conducted  at  the  Nation's  universities,  and  this  research 
is  vital  to  the  Nation's  welfare.  The  systems  NSF  and  NIH  use 
should  work  as  effectively  as  possible,  especially  considering 
shrinking  research  budgets  and  the  ever-increasing  demand  for 
technological  advances.  The  recommendations  made  in  this  report 
will  help  improve  the  scientific  performance  accountability 
systems  at  NSF  and  NIH. 
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GAO  REPORTS  AND  RELATED  STUDIES 


In  a  report  entitled  "Better  Controls  Needed  Over  Biomedical 
Research  Supported  by  the  National  Institutes  of  Health"  dated 
July  22,  1976,  GAO  recommended  that  the  Director,  NIH 


(1)  establish  specific  guidelines  outlining  what  should  be 
included  in  annual  scientific  progress  reports, 

(2)  issue  instructions  to  grant  administrators  on  how  to 
review  noncompeting  grants,  and 


(3)  incorporate  written  comments  on  the  reviews  into  the 
official  grant  files. 


NIH  took  the  following  actions  to  implement  these  recommen¬ 
dations  : 


(1)  NIH  established  minimum  requirements  for  progress  re¬ 
ports  which  include  "(a)  actual  accomplishments  toward 
meeting  project  goals,  (b)  reasons  for  not  meeting 
desired  goals,  (c)  plans  for  activities  during  the 
coming  year." 

(2)  NIH  regulations  were  strengthened  to  require  that  pro¬ 
gram  officials  review  progress  reports  for  the  above 
requirements . 


NIH  did  not  implement  our  recommendation  to  include  written  com¬ 
ments  on  progress  report  reviews  in  the  grant  files. 


In  a  report  on  NSF's  proposal  evaluation  process  entitled 
"Accountability  in  the  National  Science  Foundation's  Review 
Process  for  Grant  Awards  Needs  Strengthening"  dated  November  17, 
197°,  we  recommended  that  the  Director,  NSF: 


(1)  require  that  documentation  be  included  in  proposal 
files  to  identify  the  proposal's  critical  elements  and 
why  the  peer  reviewers  were  selected  and  (b)  show  how 
adverse  comments  and  constructive  criticisms  of  the 
proposals  which  program  officers  recommend  for  funding 
were  handled ; 

(2)  require  on  all  proposals  where  funding  is  declined  an 
explanation  of  the  peer  reviews  which  support  funding 
the  proposal,  and  why  the  proposal  was  declined;  and 

(3)  develop  internal  controls  which  assure  that  all  docu¬ 
ments  regarding  proposal  peer  review  which  NSF  permits 
researchers  to  obtain  are  sent,  when  requested  by  the 
researcher. 

NSF  disagreed  with  the  first  two  recommendations,  and  therefore, 
took  no  action  to  implement  them  stating  that  it  had  other  controls 
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in  its  process  which  provides  adequate  accountability.  NSF  still 
maintains  that  position.  On  the  third  recommendation,  NSF  said 
assuring  that  all  documents  relating  to  the  peer  review  of  a  pro¬ 
posal  are  sent  to  requesting  researchers  could  be  done  without 
developing  internal  controls.  NSF's  top  managers  were  instructed 
to  correct  any  misunderstandings  the  staff  might  have  had  on  what 
documents  are  to  be  released.  According  to  the  Director  of  NSF's 
Office  of  Audit  and  Oversight,  the  office  has  reviewed  past  re¬ 
quests  to  ensure  that  all  the  required  documents  were  sent  and 
will  "spot-check"  future  requests  for  compliance.  No  formal 
internal  controls  have  been  implemented. 

RELATED  STUDIES  BY  NSF  AND  NIH 

Both  NSF  and  NIH  have  sanctioned  reviews  of  their  peer  review 
systems.  Hie  NSF  funded  a  study  entitled  "Peer  Review  in  the 
National  Science  Foundation"  which  was  conducted  in  two  phases 
under  the  auspices  of  the  National  Academy  of  Sciences.  Hie  pri¬ 
mary  purpose  of  Phase  1,  issued  in  November  1978,  was  to  determine 
how  the  peer  review  system  functions  at  NSF  and  whether  the  system 
is  an  equitable  one  in  terms  of  fairly  evaluating  research  propo¬ 
sals.  Phase  2  is  supposed  to  more  definitively  determine  whether 
the  peer  review  system  is  equitable  by  evaluating  how  NSF  program 
officers'  selection  of  peer  reviewers  affects  grants  awarded. 

Phase  1  of  the  study  contained  several  findings  regarding 
the  relationship  of  certain  variables  to  (1)  researchers'  sci¬ 
entific  achievements,  (2)  researchers'  location  and  age,  (3)  the 
peer  reviewers'  rating  of  a  sample  of  the  researchers'  proposals, 
and  (4)  the  actions  taken  on  the  proposals  by  Foundation  program 
officers.  The  report,  however,  contains  the  following  qualifier 
regarding  the  study  results. 

"*  *  *  Where  does  the  peer  review  system  in  prac¬ 
tice  diverge  from  the  formal  statement  of  how 
peer  review  is  supposed  to  work?  Our  data  are 
well  suited  for  throwing  light  on  this  question, 
and  also  for  pointing  up  problems  with  peer  re¬ 
view.  Problems  were  revealed  in  discussions  with 
the  people  administering  the  peer  review  system, 
and  by  close  analysis  of  the  quantitative  data. 

The  research  is  not  suited  for  definitively 
answering  the  question  whether  the  peer  review 
system  is  an  'equitable*  one.  Although  our  data 
allow  us  to  speculate  usefully  on  this  question 
a  more  definitive  answer  awaits  the  completion 
of  Phase  2  of  our  research."  [Underscoring  added.] 

Although  the  study  report  contained  the  above  qualifier,  articles 
published  by  the  study  authors  in  advance  of  the  report  claimed 
the  NSF  review  system  was  "eminently  fair"  without  noting  the 
qualifier.  As  of  August  1981,  the  Phase  2  study  report  was  not  yet 
available. 
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NIH  conducted  a  two-phase  internal  review  of  its  grant  peer 
review  system  and  issued  reports  to  the  Director,  NIH,  on  its 
findings  in  1976  and  1978.  The  scope  of  the  NIH  review  was  in¬ 
tended  to  deal  with  some  issues  similar  to  those  discussed  in  this 
report  such  as  how  well  the  award  process  works  and  the  quality 
of  peer  review  as  it  relates  to  assessing  scientific  accountabil¬ 
ity.  However,  the  methodology  NIH  used  to  conduct  its  review  was 
based  largely  on  anecdotal  comments  by  and  questionnaire  informa¬ 
tion  from  peer  reviewers,  grant  applications,  and  NIH  officials. 
The  NIH  study  team,  which  was  made  up  of  various  NIH  officials, 
obtained  the  perceptions  and  opinions  of  members  of  the  scientific 
community  on  NIH's  peer  review  system.  The  study  team  concluded 
that,  based  on  these  perceptions  and  opinions,  "the  NIH  peer  re¬ 
view  system  is  and  has  been  extremely  effective  in  identifying 
biomedical  research  activities  of  high  quality."  The  study  team 
did  not  review  specific  research  grants  or  trace  the  scientific 
performance  accountability  (peer  review)  process  from  a  specific 
grant  to  the  renewal  of  that  grant. 

The  NIH  peer  review  study  team  reports  contained  numerous 
recommendations  regarding  the  operation  of  NIH's  peer  review 
process.  Most,  however,  concerned  the  administration  and/or 
management  of  the  process.  None  concerned  the  quality  of  peer 
review  or  were  related  to  improving  accountability  for  scientific 
judgments.  A  few  recommendations  were  made  regarding  substantive 
issues  of  peer  review  but  action  on  these  was  deferred  by  the  NIH 
Director  pending  further  study. 

STUDIES  BY  THE  NATIONAL 
COMMISSION  ON  RESEARCH 


The  National  Commission  on  Research  issued  two  reports  during 
1980  resulting  from  its  studies  of  scientific  accountability  over 
federally  funded  research  grants  at  the  Nation's  colleges  and  uni¬ 
versities.  The  first,  entitled  "Accountability:  Restoring  the 
Quality  of  the  Partnership"  issued  March  1980,  represented  the 
Commission's  study  of  both  accountability  for  scientific  perform¬ 
ance  and  for  financial  and  administrative  matters.  Although  this 
report  dealt  mostly  with  financial  accountability,  it  contained 
some  passages  applicable  to  scientific  accountability.  In  this 
regard  the  Commission  found  that  scientific  accountability  is 
largely  self-enforcing  without  Government  interaction  because,  in 
the  research  community,  peer  reviewers  play  a  major  role  in  decid¬ 
ing  what  work  will  be  supported,  who  shall  do  the  work,  and  what 
work  is  significant.  The  Commission  observed  that  the  conduct 
and  support  of  research  carried  out  by  universities  with  Federal 
support  should  seek  to  maintain,  strengthen,  and  support  the  com¬ 
petitive  processes  used  to  assure  integrity,  objectivity,  and 
excellence  in  the  pursuit  of  new  knowledge.  These  include  fair 
and  courageous  use  of  review  processes,  "full"  scientific  account¬ 
ability  in  the  selection  of  research  directions  and  programs  to  be 
pursued  and  supported,  and  critical  evaluation  and  dissemination 
of  research  results.  In  this  report  the  Commission  did  not  make 
specific  recommendations  regarding  scientific  accountability. 
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The  Commission's  second  report  entitled  "Review  Processes: 
Assessing  the  Quality  of  Research  Proposals"  dated  May  1980, 
specifically  concerned  the  peer  review  processes  used  by  NSF  and 
NIH.  According  to  the  report  the  Commission  conducted  a  special 
study  of  peer  review  because  the  quality  of  peer  review  is  of 
great  importance  to  the  Government  and  to  university  scientists 
and  because  peer  review  has  been  criticized  by  some  members  of 
the  Congress  and  the  academic  community.  A  Commission  subcom¬ 
mittee  made  up  of  three  Commission  members  conducted  the  study 
by  surveying  the  available  literature  and  holding  meetings  with 
and  obtaining  written  comments  from  selected  officials  from 
business.  Government,  and  the  universities.  The  Commission's 
report  concluded  that  "it  is  of  the  greatest  importance  to  the 
scientist,  particularly  in  a  situation  of  keen  competition  for 
limited  funds,  that  the  proposal  be  considered  fairly  and  criti¬ 
cally  by  the  most  competent  of  his  or  her  scientific  peers.  It 
is  equally  important  that  the  public,  represented  by  Congress 
and  the  agencies,  be  assured  that  the  money  invested  in  research 
has  funded  the  best  research  available."  The  Commission's  recom¬ 
mendations  included  (1)  that  funding  agencies  initiate  retrospec¬ 
tive  studies  on  how  effective  their  review  processes  have  been 
in  assessing  the  quality  and  effect  of  research  they  support 
which  could  suggest  improvements  in  the  predictive  accuracy  of 
the  review  of  proposals,  (2)  experimentation  by  Federal  agencies 
in  which  peer  reviewers  provide  ratings  for  several  aspects  of  a 
proposal  as  well  as  the  overall  rating,  and  (3)  funding  agencies 
make  explicit  the  nature  of  the  underlying  scientific  and  policy 
considerations  used  in  arriving  at  funding  decisions  particularly 
where  meritorious  proposals  are  clustered  around  a  narrow  range 
of  priority  ratings  which  are  resolved  by  arbitrary  numerical 
cutoffs . 
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OFFICE  OF  TNK 
DIRECTOR 


NATIONAL  SCIENCE  FOUNDATION 
WASHINGTON  D  C  20550 


July  2,  19*11 


Mr.  Morton  A.  Myers 
Director,  Program  Analysis 
Division 

General  Accounting  Office 
441  G  Street,  N.  W. 

Washington,  D.  C.  20548 

Dear  Mr.  Myers: 

1  This  is  in  response  to  your  request  for  comments  on  the  draft  report, 
"Scientific  Performance  Accountability  and  Peer  Review  Processes  for  Basic 
Research  Are  Good  But  Could  Be  Better."  We  appreciate  the  opportunity  to 
comment  on  the  draft.  As  you  note,  the  NSF  system  is  basically  in  good 
shape  but  improvements  should  always  be  sought  in  any  system.  Changes  must 
be  considered  in  the  context  of  workload  implications.  We  depend  very 
heavily  on  the  unpaid  assistance  of  thousands  of  reviewers;  we  must  be  care¬ 
ful  not  to  ask  more  of  them  than  they  are  prepared  to  give.  In  addition, 

as  you  know,  our  program  staff  has  a  very  heavy  workload;  we  can  contemplate 
additional  paperwork  only  if  the  results  warrant  the  sacrifice  of  other 
things  which  would  necessarily  have  to  be  left  undone. 

2  There  are  a  few  points  which  should  be  noted.  The  fact  that  the  NSF  peer 
review  system  has  been  studied  a  number  of  times  over  the  past  decade,  in 
some  cases  with  NSF  support  as  well  as  assistance,  demonstrates  that  we  are 
concerned  with  finding  ways  to  improve  the  system.  This  might  be  mentioned 
in  Chapter  1.  Our  use  of  our  advisory  committees  to  review  the  process  in 
each  program  over  a  several-year  period  might  also  be  mentioned. 

3  The  sample  of  grants  considered  is  quite  small;  the  statistics  are  there¬ 
fore  not  as  reliable  as  the  casual  reader  might  infer.  Also,  the  sample, 
being  limited  to  awards  to  a  few  large  institutions,  includes  an  abnormally 
large  fraction  of  investigators  who  have  multiple  support;  it  is  only  in 
the  case  of  multiple  support  that  confusion  could  arise  as  to  whether  or  not 
the  investigator 's  prior  work  was  supported  under  a  specific  previous  grant. 
We  recognize  that  indications  of  possible  Improvements  can  be  found  in  such 
a  sample  and  also  that  your  reconmendations  are  not  dependent  upon  the  sta¬ 
tistics  which  you  display  in  various  tables. 

4  The  statistics  cited  on  page  1-2  are  slightly  different  from  those  given  in 
Federal  Funds  for  Research  and  Development:  Fiscal  Years  1979,  1980  and 
1981,  Volume  XXIX  (NSF  30-318).  In  order  to  avoid  confusion,  the  source(s) 
of  your  data  might  be  cited. 
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5  Certain  points  displayed  in  Table  1  deserve  consideration  when  comparing  NSF 
and  NIH.  The  competition  for  funding  is  much  stiffer  at  NSF;  while  the  same 
number  of  proposals  are  considered  by  both  agencies,  the  number  of  awards 
made  by  NIH  is  much  broader  than  that  by  NIH.  The  decision-making  is  there¬ 
fore  more  difficult  since  NSF  must  decline  to  support  a  large  fraction  of 
truly  worthwhile  proposals.  In  addition,  the  average  length  of  an  NSF  grant 
is  less  than  that  of  one  from  NIH;  the  time  for  performance  before  submit¬ 
ting  a  renewal  request  is  therefore  shorter.  This  has  effects  on  reporting 
and  evaluating  results  from  the  previous  award. 

6  The  first  paragraph  of  Chapter  Two  can  be  read  to  say  that  the  relationship 
of  society  to  science  is  one-sided.  A  more  balanced  view  would  be  to  state 
explicitly  that  in  the  long  run  society  benefits  from  the  increased  knowl¬ 
edge  gained  as  a  result  of  the  support  provided  and  that  the  support  is  thus 
an  investment  in  the  future. 

7  Table  Two  states  that  in  both  the  NSF  systems,  peer  reviewers  do  not  deter¬ 
mine  the  scientific  merit  of  proposals.  I  assume  this  is  a  typographical 
error  since  the  function  of  peer  review  is  to  advise  the  NSF  on  the 
scientific  merit  and  probability  of  success  of  the  proposed  work.  As  noted 
in  the  draft,  as  well  as  in  other  reports,  this  advice  is  generally  the 
determining  factor  in  the  governmental  decision.  In  this  connection,  the 
first  sentence  of  the  last  paragraph  on  page  2-30  would  more  correctly 
state  the  NSF  policy  if  it  read  ".  .  .peer  reviewers  on  significant  points, 
they  must  justify.  .  ." 

8  The  first  paragraph  on  page  2-10  may  be  confusing.  The  relative  Importance 
of  criteria  vary  from  proposal  to  proposal  depending  upon  circumstances; 
thus  no  general,  rigid  requirement  would  make  sense.  As  noted,  an  expec¬ 
tation  of  competent  performance  is  a  sine  qua  non. 

9  With  regard  to  the  case  described  in  the  last  paragraph  on  page  2-12,  we 
recognize  that  such  a  situation  can  occur.  We  try  to  avoid  such  Instances 
but,  in  general,  must  rely  on  the  institution  to  correctly  identify  the 
persons  Involved  in  a  proposal. 

10  Whether  or  not  to  use  the  same  reviewers  for  a  proposal  as  were  used  on  an 
earlier  proposal  (page  2-20)  is  a  complex  question,  since  a  broad  review  is 
desirable  to  avoid  biases,  especially  on  the  question  of  the  significance 
of  the  work  proposed.  We  frequently  use  some  of  the  same  reviewers  but 
also  Include  a  large  proportion  of  new  persons  to  obtain  a  larger  sample 

of  views  and  reasons. 

11  As  noted  on  page  2-31,  NSF  staff  members,  having  been  recruited  on  the 
basis  of  scientific  expertise,  do  not  routinely  need  to  contact  the 
Investigator  concerning  negative  consents  by  reviewers.  It  might  be  noted 
that  they  do  request  responses  from  the  investigator  when  this  would  help 
to  clarify  points  which  are  important  to  a  funding  decision;  this  is 
especially  important  in  the  cases  where  the  program  officer  does  not  feel 
sufficiently  expert  in  the  sub-area  involved. 
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12  The  first  sentence  on  page  2-32  does  not  follow  from  the  question  asked. 
Given  a  limited  staff,  expertise  in  detail  in  all  sub-areas  cannot  be 
expected;  thus  a  program  officer  will  handle  proposals  for  which  he  or 
she  would  not  select  himself  or  herself  as  a  peer  reviewer  for  the  purpose 
of  obtaining  a  detailed  critique.  This  should  not  be  taken  to  imply  that 
the  program  officer  is  not  competent  to  interpret  the  reviews;  they  are, 
indeed,  qualified  to  do  so  (see  also  pages  2-46  and  4-2  where  the  same 
comment  applies) . 

13  The  second  paragraph  under  "site  visits"  on  page  2-40  should  be  clarified 
to  point  out  that  many  contacts  with  investigators  occur  at  professional 
meetings  since  this  is  a  very  cost-effective  way  of  communicating,  and 
combining  several  stops  in  one  trip  is  done  to  conserve  staff  time  and 
travel  funds. 

The  draft  report  contains  a  number  of  recommendations: 

1.  Require  that  renewal  proposal  progress  reports  identify 
the  objectives,  evidence  of  progress  toward  their  achieve¬ 
ment,  any  major  changes  in  direction  or  emphasis  and 
rationale  for  such  changes,  publications,  and/or  other 
output  resulting  from  a  researcher’s  immediately  preceding 
grant. 

14  Our  current  requirement  states  that  a  proposal  ".  .  .should  include: 
objectives  and  expected  significance;  relation  to  the  present  state  of 
knowledge  in  the  field,  to  previous  work  done  on  this  subject,  and  to 
related  work  in  progress  elsewhere.  The  statement  should  outline  the 
general  plan  of  work,  including  the  broad  design  of  experiments  to  be 
undertaken  and  an  adequate  description  of  experimental  methods  and  proce¬ 
dures  .  ."  For  renewal  proposals  we  require  "a  summary  of  progress  to  date 
and  its  relation  to  the  proposed  work."  We  interpret  the  GAO  recommendation 
to  suggest  a  greater  distinction  between  short-term  objectives  and  long- 
range  goals  and  to  suggest  that,  where  an  investigator  has  multiple  support, 
renewal  proposals  should  indicate  more  clearly  which  accomplishments  were 
achieved  under  the  predecessor  grant (s) .  We  will  reexamine  our  current 
guidelines  to  proposers  and  make  such  changes  as  are  necessary  to  clarify 
these  points. 

2.  Establish  a  requirement  that  peer  reviewers  be  asked  when 
reviewing  renewal  proposals  to  specifically  comment  on  a 
researcher's  performance  on  the  immediately  preceding  grant. 


15  We  currently  ask  reviewers  to  cotmnent  on  an  investigator's  recent  accomplish¬ 
ments  which,  in  many  cases,  is  sufficient.  As  noted  earlier,  the  nature  of 
the  GAO  sample  resulted  in  an  abnormally  large  fraction  of  cases  of  multiple 
support.  The  changes  noted  under  the  first  recommendation  will  enable  us  to 
ask  reviewers  to  consent  more  explicitly  in  cases  of  multiple  support  and 
we  will  do  so. 

3.  Require  that  panel  peer  review  deliberations  be  documented 
along  the  lines  of  the  NIH  peer  review  group  summary  state- 
ment. 
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16  We  believe  chat  the  GAO  has  overlooked  a  major  difference  between  NSF  and  NIH 
in  making  this  recommendation.  Almost  all  basic  research  proposals  reviewed 
by  NSF  panels  are  also  reviewed  ad  hoc.  These  reviews  as  well  as  written  indi¬ 
vidual  reviews  by  one  or  more  panel  members,  are  available  verbatim  to  proposers. 
I  also  note  that  the  GAO  sample  was  drawn  from  awards  made  well  before  1978  when 
the  GAO  staff  drew  our  attention  to  the  fact  that  some  panels  did  not  produce 
written  summaries.  We  now  require  such  summaries,  but,  because  of  the  more 
detailed  comments  of  ad  hoc  reviewers  (including  some  panel  members)  are  avail¬ 
able,  these  summaries  can  be,  and  are,  brief.  A  more  extensive  summary  would 
not  be  a  productive  use  of  staff  time. 

4 .  Require  that  peer  review  comments  be  automatically  sent 
to  researchers . 


17  The  routine  forwarding  of  peer  review  comments  is  already  more  extensive  than 
the  report  implies.  All  divisions  in  the  Biological,  Behavioral,  and  Social 
Sciences  Directorate  now  follow  this  practice  as  do  a  number  of  other  units. 

We  have  been  discussing  extending  this  procedure  and  will  now  make  it  universal. 

5 .  Require  proposals  to  identify  the  research  objectives  to  be 
undertaken  during  the  grant  period. 

18  Our  response  is  contained  in  the  response  to  the  first  recommendation. 

6.  Require  that  proposals  for  new  projects  include  evidence 
of  progress  from  the  prior  grant(s). 

19  Our  response  is  contained  in  the  response  to  the  first  recommendation. 

7.  Ensure  that,  when  researchers  seek  funding  for  new  projects 
peer  reviewers  are  furnished  the  final  technical  report  and 
listing  of  publications  from  the  prior  grant (s).  when  avail¬ 
able. 

20  We  currently  require  the  investigators  to  provide  a  list  of  their  recent 
publications.  Final  technical  reports  are  usually  not  available  until  well 
after  a  renewal  proposal  is  reviewed;  similarly,  there  is  a  considerable  delay 
between  the  time  work  is  done  and  the  appearance  of  publications  describing  it. 
In  addition  to  the  changes  discussed  above,  we  will  ask  that  those  seeking 
renewal  support  specifically  identify  publications  resulting  from  the  work 
under  the  predecessor  grant(s). 

8 .  Assure  more  systematic  and  uniform  review  of  annual  progress 
reports  by  the  program  officers. 

21  Many  of  our  awards  are  made  as  "continuing"  grants;  these  are  approved  for, 
typically,  three  years  but  are  funded  one  year  at  a  time.  In  order  to  receive 
the  annual  Increment  of  funds,  a  progress  report  is  required  and  is  reviewed 
by  the  program  officer  prior  to  recommending  the  funding  increment.  Other 
annual  reports  and  copies  of  publications  resulting  from  grants  are  read  by 
the  staff  when  they  arrive.  We  will  reexamine  our  practice  to  see  what  changes 
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might  be  appropriate  but  cannot  agree  that  the  Imposition  of  more  paperwork  on 
our  busy  staff  would  be  beneficial. 

9.  Establish  more  specific  guidelines  regarding  the  extent 
to  which  researchers  can  change  grant  objectives  without 
prior  agency  approval ■ 

22  This  is  a  complicated  matter.  In  contrast  to  NSF,  NIH  has  some  mission  interests 
even  in  the  basic  research  that  they  support.  As  GAO  has  noted,  research  plans 
must  change  as  new  knowledge  develops.  The  question  of  where  in  the  continuum 
from  a  minor  change  of  instrumental  technique  to  a  complete  change  of  objective 
we  should  require  prior  approval  is  not  an  easy  question  to  answer;  it  is  not 
clear  that  a  complete  answer  can  be  found.  Much  must  necessarily  be  left  to 
judgments.  We  will  study  this  question  to  see  if  we  can  define  Improved  guid¬ 
ance  so  that  the  individual  judgments  may  be  more  consistent. 

Thank  you  for  the  opportunity  to  comment  on  the  draft  report.  If  you  have  any 
questions,  please  contact  Dr.  J.  H.  Fregeau,  Director,  Office  of  Audit  and 
Oversight. 


Sincerely  yours, 


V  John  B.  Slaughter 
Director 
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GAO  RESPONSE  TO  NSF  COMMENTS 


The  number  of  the  responses  below  correspond  to  the  numbered 
paragraphs  of  the  July  2,  1981,  letter  from  John  B.  Slaughter, 
Director  of  the  National  Science  Foundation. 

1.  Our  proposed  recommendations,  with  the  exception  of  #3 
noted  on  page  67,  would  not  impose  an  additional  workload  on 
NSF's  program  officers.  In  fact,  implementing  the  recommendations 
should  reduce  the  workload  in  that  proposal  evaluation  should  be 
easier  and  thus  less  time-consuming.  The  proposed  recommendation 
(#3)  that  would  slightly  increase  the  workload  has  been  changed 

to  reflect  this  and  now  asks  only  what  is  considered  the  minimum 
information  necessary.  Further  response  on  this  point  is  given 
on  page  73 . 

2.  We  recognize  that  NSF  has  supported  and  assisted  a  num¬ 
ber  of  studies  of  its  peer  review  system.  However,  in  chapter  1 
and  appendix  I,  we  note  those  studies  that  are  directly  applicable 
to  this  report.  Studies  other  than  those  we  mention  have  been 
done  of  NSF's  peer  review  system,  but  none  directly  involves  the 
issue  of  this  report — scientific  performance  accountability. 

Also,  almost  all  studies  of  NSF's  peer  review  system  were  initi¬ 
ated  or  requested  by  congressional  committees  and  not  by  NSF. 
Regarding  NSF's  advisory  committees,  we  are  presently  evaluating 
the  advisory  committees '  role  in  reviewing  the  peer  review  process 
in  each  NSF  program. 

3.  We  selected  our  sample  of  grants  from  major  research 
universities  because  these  institutions  receive  the  bulk  of  NSF 
grant  funds.  We  agree  that  the  problems  noted  in  this  report  are 
greater  when  researchers  have  more  than  one  grant.  Since  research¬ 
ers  with  multiple  grants  are  concentrated  at  large  institutions 
and  since  these  institutions  get  the  bulk  of  NSF  funds,  our  sample 
represents  the  area  where  the  greatest  concern  is  with  assuring 
scientific  accountability  on  basic  research  grants. 

However,  we  disagree  with  NSF's  contention  that  our  recom¬ 
mendations  only  apply  to  researchers  with  multiple  grants.  Con¬ 
fusion  regarding  a  researcher's  prior  work  can  arise  when  the 
researcher  had  only  one  previous  grant  (see  example  discussed  on 
page  27  of  this  report).  We  intend  that  the  recommendations 
in  this  report  apply  to  proposals  from  all  researchers  either  for 
renewed  support  of  an  existing  project  or  support  of  a  new  project 
which  immediately  follows  an  existing  project. 

4.  The  statistics  cited  on  page  1  regarding  Federal  funds 
provided  to  colleges  and  universities  were  derived  from  the  NSF 
and  NIH  fiscal  year  (FY)  1981  budget  presentations  and  the  docu¬ 
ment  NSF  refers  to  in  its  letter  (NSF  30-318).  However,  the 
statistics  cited  in  NSF  30-318  to  which  NSF  refers  in  its  let¬ 
ter  grossly  understate  NIH's  basic  research  funding  level.  NIH 
confirmed  the  statistics  we  use  in  the  report.  NSF's  statistics 
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reflect  those  given  in  NSF  30-318,  which  shows  total  Federal  basic 
research  funds  to  colleges  and  universities  to  be  about  $2.3  bil¬ 
lion.  NSF  30-318  shows  the  NIH  portion  to  be  about  $1  billion  in 
FY  1980  whereas  NIH  budget  documents  show  it  as  $1.6  billion  for 
FY  1980.  The  $2.3  billion  total  shown  in  NSF  30-318,  when  added 
to  the  $.6  billion  underestimate  for  NIH,  add  to  $2.9  billion — the 
figure  we  use. 

5.  The  competition  for  funding  at  NSF  varies  greatly  among 
the  various  programs.  Some  programs,  such  as  physics,  fund  over 
70  percent  of  all  proposals  received.  Some  life  science  programs 
fund  only  20  percent.  Of  the  proposals  reviewed  in  FY  1980  NSF 
funded  about  49  percent;  NIH,  about  62  percent.  Both  agencies 
must  decline  many  tiuly  worthwhile  proposals.  The  average  length 
of  NSF  grants,  while  still  shorter  than  NIH's,  is  increasing  as 
more  3-year  continuing  grants  are  awarded.  However,  with  the 
shorter  term  grants  it  is  equally  if  not  more  important  to  provide 
the  peer  reviewers  and  program  officers  with  sufficient  data  which 
will  permit  an  effective  evaluation.  Although  the  performance 
periods  might  be  shorter,  the  need  to  adequately  assess  scientific 
performance  accountability  is  just  as  great,  if  not  greater. 

6.  No  response  is  required. 

7.  We  revised  the  wording  in  table  2  to  reflect  that  the 
peer  reviewers  do  not  make  the  decisions  on  proposals  although 
NSF  *  s  peer  reviewers  do  advise  NSF  on  proposals'  scientific  merit. 
NSF's  suggested  wording  in  the  last  sentence  was  added  to  this 
final  report. 

8.  We  are  not  suggesting  that  rigid  criteria  be  used  for 
every  proposal.  However,  the  criteria  used  by  NSF  appear  to  place 
more  weight  on  a  researcher's  track  record  than  on  the  scientific 
merit  of  the  proposed  work. 

9.  No  response  is  required. 

10.  Our  sample  data  on  page  26  show  that  NSF  does  not 
frequently  use  the  same  reviewers  to  review  successive  proposals. 
Our  sample  statistics  show  that  for  37  percent  of  such  proposals 
none  of  the  same  reviewers  were  used  to  review  the  renewal  propo¬ 
sal  who  also  reviewed  the  sample  grant  proposal  and  that  on  an¬ 
other  52  percent  only  one  reviewer  who  had  reviewed  the  previous 
proposal  also  reviewed  the  renewal  proposal.  We  do  not  intend 
that  NSF  use  all  the  same  reviewers  on  renewal  proposals  who  re¬ 
viewed  the  previous  proposals.  However,  because  of  high  program 
officer  turnover  at  NSF,  more  continuity  of  reviewers  from  one 
proposal  to  the  next  is  needed  to  improve  proposal  review  than 
what  we  showed  in  our  sample. 

11.  Our  review  of  the  sample  grant  files  did  not  show  evi¬ 
dence  that  program  officers  were  contacting  researchers  when 
negative  comments  were  made  by  the  peer  reviewers,  so  we  asked 
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program  officers  whether  they  routinely  contact  researchers 
about  negative  comments. 

12.  Wording  of  the  material  on  page  32  was  changed  to  bet¬ 
ter  represent  the  responses  to  the  question.  We  do  not  imply  on 
page  32  that  program  officers  cannot  properly  interpret  the 
peer  reviews  when  they  lack  the  expertise  to  be  peer  reviewers. 

We  agree  with  NSF  that  program  officers  do  not  need  to  be  peer 
reviewers  to  properly  judge  the  peer  reviews  of  a  proposal.  The 
report  merely  stresses  the  apparent  inportance  of  peer  reviewer 
advice  to  the  program  officers.  The  information  on  pages  41 

and  54  has  no  relevance  to  that  on  page  32.  Report  page  41 
states  that  some  program  officers  do  not  have  the  expertise  to 
evaluate  some  grant  results — which  is  not  disputed  by  NSF. 

Report  pages  41  and  54  do  not  suggest  or  imply  that  program 
officers  who  do  not  have  the  expertise  to  be  peer  reviewers  on  a 
proposal  also  cannot  interpret  the  peer  reviews  that  are  received. 
Report  page  41  suggests  that  when  program  officers  cannot  be 
peer  reviewers  on  a  grant,  they  also  cannot  evaluate  the  grant 
results,  since  they  needed  to  be  a  peer  reviewer  on  the  grant. 

13.  The  changes  that  are  indicated  have  been  made. 

14.  Our  recommendation  regarding  the  progress  report  sec¬ 
tion  in  renewal  proposals  is  intended  to  have  all  researchers 
show  the  objectives,  evidence  of  progress  toward  their  achieve¬ 
ment,  any  major  changes  in  direction  or  enphasis  and  the  reasons, 
and  the  publications  or  other  output  from  researchers’  immediately 
preceding  grants.  While  the  problem  is  greater  with  researchers 
who  have  multiple  grants,  it  also  occurs  when  researchers  have 
only  one  grant.  NSF's  interpretation  of  only  clarifying  short- 
versus  long-term  objectives  for  researchers  with  multiple  support 
does  not  solve  the  problem.  All  researchers  seeking  continued 
support  for  an  ongoing  project  need  to  restate  the  objectives  of 
their  prior  grant  in  the  renewal  proposal  along  with  progress 
made  and  the  resulting  output.  This  requirement  is  particularly 
crucial  for  researchers  having  nultiple  support  (see  chapter  2). 
When  a  researcher  has  two  or  more  ongoing  grants  and  submits  a 
proposal  to  renew  one  of  them,  unless  the  proposal  clearly  shows 
the  objectives,  etc.,  of  the  grant  for  which  renewal  is  sought, 
the  peer  reviewers  and  program  officers  can  be  confused  as  to 
what  work  goes  with  what  grant.  Merely  asking  that  a  greater 
distinction  be  made  between  long-  and  short-term  objectives  does 
not  rectify  the  situation.  Our  recommendation  is  intended  to 
assure  that  pear  reviewers  and  program  officers  have  more  specific 
information  on  the  prior  grant  included  in  all  renewal  proposals 
regarding  research  objectives,  progress  made,  changes  in  direc¬ 
tion,  and  output  unler  the  inmediately  preceding  grant  for  which 
renewal  is  sought.  More  specific  information  wil»  enable  review¬ 
ers  and  program  officers  to  better  assess  a  researcher’s  perform¬ 
ance  on  a  specific  prior  grant  for  which  renewal  is  sought  and 

the  significance  of  that  progress--wh ich  should  improve  the  sci¬ 
entific  performance  accountability. 
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This  recommendation  requires  no  additional  staff  time  or 
time  of  the  peer  reviewers  to  carry  out.  in  fact,  implementing 
this  recommendation  could  save  review  time  by  providing  informa¬ 
tion  that  could  make  evaluation  easier,  and  thus  less  time- 
consuming.  Restating  specific  prior  grant  objectives  and  showing 
progress  male  an  1  any  output  that  resulted  from  that  specific 
grant  will  make  it  easier  for  pear  reviewers  and  program  officers 
to  evaluate  all  proposals — those  from  researchers  with  multiple 
grants  as  well  as  those  from  researchers  with  only  one  prior 
grant.  Better  and  more  timely  proposal  evaluations  should  improve 
scientific  performance  accountability  for  all  NSF  grants.  Chapter 
2  contains  examples  showing  the  reasons  for  this  recommendation 
and  why  it  is  needed  for  single,  as  well  as  multiple,  funded 
researchers . 

15.  NSF  states  that  they  will  ask  reviewers  to  comment  more 
explicitly  in  cases  of  multiple  support.  While  this  will  help 
solve  part  of  the  problem  which  pro  noted  the  recommendation,  NSF's 
intended  actions  are  not  sufficient.  We  intend  the  recommendation 
to  apply  to  all  renewal  proposals  including  those  from  researchers 
who  do  not  have  multiple  support.  Peer  reviewers  should  always 
specifically  comment  on  the  immediate  past  performance  of  the  re¬ 
searcher  requesting  continued  support.  However,  sometimes  they 

do  not.  As  noted  on  page  21,  in  only  22  percent  of  the  renewal 
cases  in  our  sample  were  there  evaluative  comments  about  the  re¬ 
searcher's  performance  under  the  immediately  preceding  grant. 

When  the  peer  reviewers  specifically  comment  on  the  researcher's 
performance  under  the  immediately  preceding  grant,  a  better 
assessment  of  the  merits  of  continuing  funding  is  obtained  by  the 
program  officers. 

16.  We  changed  our  recommendation  to  reflect  NSF's  views. 

NSF  dil  not  believe  that  it  should  docunent  its  summary  of  panel 
peer  review  deliberations  along  the  lines  of  the  NIH  peer  review 
group  summary  statement  because,  NSF  noted,  almost  all  its  propo¬ 
sals  that  are  panel  reviewed  also  receive  ad  hoc  review.  As  a 
result,  NSF  believes  that  because  of  the  ad  hoc  reviews,  which 
are  written,  as  well  as  the  written  reviews  of  individual  panel 
members,  the  panel  summaries  can  be,  and  are,  brief.  At  NIH, 

the  peer  review  group  summary  statement  provides  the  primary 
evidence  showing  the  results  of  peer  review.  Written  reviews  by 
individual  peer  reviewers  are  not  available. 

NSF  stated  that  for  most  panel  reviewed  proposals  some  writ¬ 
ten  peer  reviews  are  routinely  available,  which  largely  eliminates 
the  need  for  a  panel  summary  similar  to  NlH’s.  The  thrust  of 
NSF's  concern  is  valid.  However,  we  believe  the  individual  ad 
hoc  and  panel  peer  reviewers'  written  reviews  together  with  the 
panel  summary  should,  collectively,  show  the  peer  reviewers' 
reasons  for  their  recommendation,  a  description  of  the  research, 
a  critique  of  the  research,  comments  on  the  researcher's  qualifi¬ 
cations,  competence,  resources  and  facilities,  and  a  budget  eval¬ 
uation.  When  the  individual  reviews  do  not  show  this,  the  panel 
summary  should.  The  recommendation  in  this  report  regarding 
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NSF's  panel  summaries  reflects  NSF's  concern  and  our  belief  that 
panel  summaries,  together  with  other  available  written  evidence, 
should  show,  as  a  minimum,  information  on  the  items  noted  above. 

17.  The  changes  that  are  indicated  have  been  made. 

18.  NSF  interpreted  this  recommendation  to  mean  that  pro¬ 
posals  from  researchers  with  only  multiple  support  should  show 
the  short-term  objectives  applicable  to  the  specific  grant  period. 
However,  the  recommendation  is  intended  to  require  that  all  pro¬ 
posals  from  all  researchers  contain  the  research  objectives  the 
researcher  plans  to  work  on  during  the  specific  period  of  time 
that  funding  is  sought.  We  intend  that  specific  grant  period 
objectives  be  included  in  both  renewal  as  well  as  new  project 
proposals  and  in  proposals  from  researchers  with  single  as  well 

as  multiple  support.  The  purpose  of  the  recommendation  is  to 
facilitate  better  evaluation  by  the  peer  reviewers  and  program 
officers  of  the  progress  made  under  the  grant  when,  and  if, 
renewal  is  sought.  Knowledge  of  the  specific  grant  period  ob¬ 
jectives  will  facilitate  better  evaluation. 

19.  Hiis  proposed  recommendation  is  not  intended,  as  NSF 
suggests,  to  apply  only  to  researchers  with  multiple  grants. 

This  recommendation  is  applicable  to  all  proposals  for  new  proj¬ 
ects  when  the  researcher  had,  or  has,  another  project  in  the 
same  line  of  research,  even  if  the  researcher  only  had,  or  has 
one  such  project.  The  recommendation  is  especially  important, 
however,  for  researchers  with  more  than  one  project.  The  recom¬ 
mendation  is  intended  to  assure  that  there  be  evidence  in  the 
new  project  proposal  showing  progress  made  on  the  prior  grant 
even  though  the  prior  grant  involved  a  different  research  project. 
As  shown  in  chapter  2,  a  researcher  can  continually  propose  new 
projects  and  not  necessarily  be  held  accountable,  at  least  in 

the  short  term.  This  recommendation  will  better  assure  that 
peer  reviewers  and  program  officers  have  evidence  in  the  new 
project  proposal  of  the  researcher's  immediate  past  grant  progress 
and  performance.  As  noted  in  chapter  2,  this  is  not  always  the 
case  in  that  the  peer  reviewers  could  not  adequately  assess  the 
new  project  proposal  without  some  evidence  of  the  researcher's 
immediate  past  work,  which  was  not  specifically  stated.  Also, 
because  a  proposal  for  a  new  project  will  often  be  submitted 
before  results  of  the  prior  grant  are  published,  the  prior  grant 
results  might  not  be  known  and  evidence  in  the  new  project  propo¬ 
sal  might  be  the  best  (and  perhaps  only)  way  peer  reviewers  or 
program  officers  have  of  assessing  prior  grant  performance.  As 
a  result,  this  recommendation  applies  to  all  new  project  proposals 
whether  from  researchers  with  single  or  multiple  support,  recog¬ 
nizing  that  it  is  most  important  to  have  this  information  in 
those  proposals  from  researchers  with  multiple  grants. 

20.  No  response  is  required. 

21.  NSF  agreed  to  reexamine  its  practice  to  see  what  changes 
might  be  appropriate,  but  did  not  agree  that  the  imposition  of 
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22.  no  response  is  required. 
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DEPARTMENT  OF  HEALTH  *.  HUMAN  SERVICES 


Office  of  Inspector  General 


Washington,  D  C  20201 


9  JUL 


Mr.  Gregory  J.  Ahart 
Director,  Human  Resources 
Division 

United  States  General 
Accounting  Office 
Washington,  D.C.  20548 

Dear  Mr.  Ahart: 

The  Secretary  asked  that  I  respond  to  your  request  for  our 
comments  on  your  draft  report  entitled,  "Scientific  Performance 
Accountability  and  Peer  Review  Processes  for  Basic  Research 
Are  Good  But  Could  Be  Better."  The  enclosed  comments  represent 
the  tentative  position  of  the  Department  and  are  subject 
to  reevaluation  when  the  final  version  of  this  report  is 
received. 

We  appreciate  the  opportunity  to  comment  on  this  draft 
report  before  its  publication. 


Sincerely  yours, 


✓p^^Ri^mard  P.  Kusserow 
Inspector  General 


Enclosure 
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COMMENTS  OF  THE  DEPARTMENT  OF  HEALTH  AND  HUMAN  SERVICES  ON  THE 
GENERAL  ACCOUNTING  OFFICE  DRAFT  REPORT  "SCIENTIFIC  PERFORMANCE 
ACCOUNTABILITY  AND  PEER  REVIEW  PROCESSES  FOR  BASIC  RESEARCH  ARE 
GOOD  BUT  COULD  BE  BETTER" 


GAO  Recommendation 


We  recommend  that  the  Director  of  NIH: 

--Require  that  proposals  for  new  projects  include  evidence  of  progress 
from  the  prior  grant(s). 

--Ensure  that,  when  researchers  seek  funding  for  new  projects,  peer 
reviewers  are  furnished  the  final  technical  report  and  listing  of 
publications  from  the  prior  grant(s),  when  available. 

Department  Comment 

We  concur  with  the  intent  of  these  recommendations.  However,  any  decision 
to  implement  GAO's  recommended  actions  will  be  delayed  until  NIH's 
Review  Policy  Coirmittee  (RPC),  which  serves  in  an  advisory  capacity  to 
the  Director,  NIH,  reviews  the  practices  and  procedures  involving  information 
provided  to  peer  reviewers.  These  items  will  be  included  on  the  agenda 
for  the  RPC's  September  1981  meeting.  RPC  will  provide  its  recommendations 
to  the  Director,  NIH,  in  January  1982.  If  it  is  determined  that  changes 
are  needed,  NIH  will  take  action  to  implement  those  items  noted  in  the 
GAO  recommendations  by  April  1982. 

GAO  Recommendation 


We  recommend  that  the  Director  of  NIH  assure  more  systematic  and  uniform 
review  of  annual  progress  reports  by  the  program  officers. 

Department  Comments 


We  concur.  We  will  make  a  thorough  examination  of  current  practices  at 
NIH  and  use  this  information  to  develop  a  uniform  process  for  receipt 
and  evaluation  of  progress  reports  by  January  1982. 

GAO  Recommendation 


We  recommend  that  the  Director  of  NIH  establish  more  specific  guidelines 
regarding  the  extent  to  which  researchers  can  change  grant  objectives 
without  prior  agency  approval. 

Department  Comments 

We  concur.  We  recognize  the  need  to  be  as  specific  as  possible  regarding 
grant  objectives  and  we  will  develop  more  definitive  guidelines  by 
January  1982. 
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Page  2 


Technical  Comments 


1  In  general,  the  report  fairly  presents  the  issues  involved.  However, 
the  words  "secrecy"  and  "anonymity'  used  to  describe  NIH  peer  review 
process  could  be  misleading.  In  recent  years,  NIH  has  made  changes  in 
its  peer  review  process  making  it  significantly  more  open.  For  instance, 
complete  rosters  of  all  appointed  and  initial  review  group  members  are 
made  available  to  the  public  semiannually.  Further,  summary  statements 
of  the  grant  initial  review  groups,  which  include  deliberations  and 
recommended  actions  on  each  application,  are  automatically  given  to  the 
applicant  Principal  Investigator.  In  our  opinion,  these  and  other 
changes  provide  an  openness  to  the  peer  review  process  that  is  not 
adequately  conveyed  by  the  words  "secrecy"  and  "anonymity." 

2  In  Table  2  on  page  2-3  of  the  draft  report,  the  process  "award  decision 
based  on  scientific  merit,  made  by  program  officer"  is  shown  as  "no"  for 
NIH.  This  should  be  "yes"  because  program  officers  at  NIH  are  involved 
in  the  decision  to  make  an  award  based  on  scientific  merit  as  evaluated 
by  peer  review. 


78 


APPENDIX  III 


APPENDIX  III 


GAO .RESPONSE  TO  HHS  COMMENTS 

No  response  is  required  for  the  first  two  pages  of  the  HHS 
comments.  HHS  concurred  with  all  our  recommendations  to  the 
Director,  NIH,  subject  to  review  by  NlH's  Review  Policy  Committee. 
Our  response  to  paragraphs  1  and  2  on  page  78  follow. 

1.  Additional  wording  has  been  added  to  chapter  1  to  clarify 
the  words  "secrecy"  and  "anonymity."  Although  the  names  of  NIH 
peer  reviewers  who  serve  on  the  review  groups  are  made  public  and 
summary  statements  are  automatically  given  to  the  researchers, 

the  NIH  peer  review  process  (as  well  as  NSF's)  is  conducted  in 
relative  secrecy  and  the  anonymity  of  individual  peer  reviewers 
who  review  specific  proposals  i3  maintained.  Peer  review  group 
meetings  are  closed  to  the  public.  Individual  peer  reviewers’ 
comments  on  specific  NIH  proposals  are  not  disclosed  or  made 
available  outside  the  group.  The  researchers  only  receive  the 
sumnary  statement  prepared  on  each  proposal.  The  summary  state¬ 
ment  does  not  identify  a  specific  peer  reviewer's  comments.  As 
a  result,  although  the  names  of  group  members  are  made  public, 
individual  group  members'  opinions  of  specific  proposals  are  not 
disclosed,  even  to  the  researcher. 

2.  The  wording  in  table  2  was  changed  to  show  that  NIH 
program  officers  cannot  award  a  proposal  as  a  grant  unless  the 
peer  reviewers  have  first  approved  the  scientific  merit  of  the 
proposal . 
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UNIVERSITY  OF  WISCONSIN-MADISON 


CHANCELLOR 

EUscom  Hall  •  S00  Lincoln  Drive 
Madison,  Wisconsin  53706 
608-262-9946 


June  24,  1981 


Mr.  Osmund  T.  Fundingsland 
Associate  Director 
Science  and  Technology 
U.  S.  General  Accounting  Office 
Washington,  D.  C.  20548 

Dear  Mr.  Fundingsland: 

1  I  wish  to  comment  upon  the  excerpts  from  the  draft  General  Accounting 
Office  report  on  NSF  and  NIH  systems  for  scientific  performance  account¬ 
ability. 

2  Although  various  universities  may  exhibit  some  differences  In  detail, 
the  description  of  the  proposal  submission  process  is  generally  accurate. 

It  is  appropriate  that  the  review  of  proposals  for  scientific  merit  and 
need  or  relevance  of  the  proposed  research  be  carried  out  externally. 

3  The  second  section  of  the  report,  headed  "Universities  do  not  monitor 
scientific  progress"  raises  several  questions,  and  contains  assumptions 
which  we  must  challenge.  A  distinction  must  be  made  between  grants  for 
basic  research,  and  contracts  for  the  performance  of  specific  research 
tasks.  Performance  schedules  and  milestones  in  general  are  found  only 

in  the  case  of  contracts.  A  very  small  fraction  of  NSF  and  NIH  awards 
to  the  University  of  Wisconsin-Mad Ison  come  in  the  form  of  contracts; 
however,  in  these  cases  we  do  monitor  where  appropriate  and  feasible 
the  achievement  of  these  milestones. 

4  Research  grants,  on  the  other  hand,  are  best  monitored  at  the  comple¬ 
tion  of  the  project  period.  Optimal  strategy  for  performing  the  research 
may  dictate  non-uniform  effort  on  the  grant  over  the  grant  period.  This 
is  generally  assumed  in  the  funding  agencies.  It  is  Important  to  recognize 
that  both  NSF  and  NIH  grants  require  annual  reports  to  the  agencies,  as 
well  as  periodic  competitive  renewals.  Given  the  lead  time  necessary 

for  annual  renewals,  reports  to  the  agencies  must  be  prepared  from  six 
to  eight  months  into  the  grant  period.  This  provides  an  effective 
nechanism  for  monitoring  scientific  progress.  Indeed,  for  the  reason 
cited  above,  even  this  may  be  too  severe  a  monitoring  mechanism. 

5  Both  the  effort  reporting  system  and  general  university  fiscal 
controls  provide  effective  mechanisms  to  insure  sound  management  practices 
in  the  administration  of  grants. 
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Mr.  Osmund  T.  Fundingsland  -2- 


June  24,  1981 


6  Finally,  1  must  take  strong  issue  with  the  statement  in  Conclusions 

regarding  performance  accountability  "Even  the  universities,  which 
theoretically  are  in  a  position  to  provide  it,  practically  cannot  and 
do  not  because  of  the  'academic  freedom'  environment  and  the  lack  of 
expertise  about  the  research  being  performed."  "Academic  freedom," 
of  course,  does  not  enter  into  questions  of  performance  accountability, 
nor  is  it  fair  to  assume  a  lack  of  expertise.  Rather,  as  explained 
above,  the  appropriate  measure  of  performance  accountability  is  being 
provided  through  agency  review,  which,  being  external,  is  probably 
preferable. 


jrs 

xc:  Associate  Dean  Marvin  E.  Ebel 
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GAO  RESPONSE  TO  WISCONSIN  COMMENTS 

The  numbers  of  the  responses  below  correspond  to  the  numbered 
paragraphs  of  the  June  24,  1931,  letter  from  the  University  of 
Wi  scons in-Madi so n . 

1.  No  response  is  required. 

2.  No  response  is  required. 

3.  This  section  of  the  report  does  not  contain  any  assump¬ 
tions.  It  only  states  the  requirements  of  NSP  and  NIH  grant  pol¬ 
icy  manuals.  The  report  also  is  clearly  directed  at  only  research 
grants.  The  report  does  not  concern  contracts  and  we  state  this 
in  numerous  places. 

4.  This  paragraph  confuses  monitoring  with  evaluation. 
Monitoring  implies  an  activity  that  occurs  throughout  the  project 
period.  Evaluation  occurs  at  the  end  of  the  period.  The  rest 

of  the  paragraph  apparently  reflects  the  University's  view  on 
when  monitoring  should  occur.  However,  both  NSF  and  NIH  grant 
policy  maauals  require  that  universities  monitor  the  scientific 
performance  aspects  of  research  grants  (3ee  page  39).  Research¬ 
ers  we  interviewed  stated  that  their  universities  do  not  monitor 
their  research  (see  page  39).  We  did  not  interview  university 
administrators  on  this  subject. 

5.  Agency  grant  policy  manuals  also  require  that  universi¬ 
ties  be  responsible  for  the  scientific  as  well  as  administrative 
aspects  of  research  grants. 

6.  Academic  freedom  is  not  an  issue  in  scientific  perform¬ 
ance  accountability  at  the  universities  because  all  but  one  of 
the  researchers  GAO  interviewed  said  their  universities  do  not 
monitor  the  technical  aspects  of  their  research  nor  do  the  univer¬ 
sities  review  research  proposals  for  technical  adequacy.  Re¬ 
searchers  also  said  that  in  many  instances  the  universities  do 
not  get  involved  in  the  substance  of  their  research  at  any  level 
largely  because  few  if  any  other  persons  at  the  university  have 
the  expertise  to  monitor  the  scientific  aspects  of  the  individual 
researcher's  research. 
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STANFORD  UNIVERSITY 

STANFORD,  CALIFORNIA  94505 


SPONSORED  PROJECTS  OFFICE 
Encina  Hale 
660  Atcutuo  Wav 


July  6,  1981 


Telephone  (415)  497-288) 
Teles  148  402  Stanlrd  STNU 


Osmund  T .  Fundings  land 
Associate  Director, 

Science  and  Technology 

United  States  General  Accounting  Office 

Washington,  D.C.  20548 

Dear  Dr.  Fundingsland: 

This  responds  to  your  June  9  letter  to  President  Kennedy  asking  for 
review  and  comment  on  Pages  1-3  your  draft  Report  PAD  61-29  on 
scientific  performance  accountability  systems  for  NIH  and  HSF  basic 
research  granta. 

He  find  ourselves  in  disagreement  with  the  findings  and  conclusions 
regarding  university  proposal  review  and  grant  accountability  with 
respect  to  scientific  performance. 

For  the  sake  of  convenience  and  facility  of  communication,  I  have  framed 
our  comments  in  the  form  of  revisions  to  the  draft  report’s  text. 

First  page  headed,  PROPOSAL  SUBKXSSIOH  —  third  paragraph,  first 
line:  insert  "centrally"  between  "review"  and  "proposals." 

Page  2  —  replace  first  sentence  aB  follows; 

"While  the  universities  we  visited  do  not  have  central  technical 
review  boards  to  monitor  scientific  progress  despite  NSF's  and 
HIH's  policies  that  they  be  responsible  and  accountable  for  grant 
activities;  in  fact,  they  do  accept  those  responsibilities. 
Rather,  they  place  the  scientific-technical  accountability 
performance  responsibility  on  the  senior  ("Principal") 

investigator  (research  director)  named  as  responsible  in  the  award 
notice.  By  doing  ao,  by  controlling  the  eligibility  of  personnel 
for  that  designation  stringently,  and  by  making  the  academic 
department  heads  and  academic  deans  responsible  for  the  oversight 
of  the  academic-research  programs  in  their  departments  and 
schools,  they  believe  they  obtain  adequate  oversight  of  this 
aspect  of  the  accountability  requirements. 

"In  addition,  schools  we  visited  have  appointed  a  senior  academic 
officer  with  overall  responsibility  for  the  proper  conduct  of  the 
research-graduate  education  programs  across  the  institution  and 
for  the  Implementation  and  review  of  appropriate  policies  to 
assure  effective  academic  controls.  These  officers  are  frequently 
assisted  by  professional  staff  and  one  or  more  academic  policy 
advisory  groups  which  include  members  of  the  faculty  and 
administrative  staff  officers. 
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"Central  administrative  staffs  maintain  the  official  records  for 
administrative  accountability  matters  and  have  oversight 
responsibility  for  the  effectiveness  and  integrity  of  the 
records . " 

Page  3  -  CONCLUSIONS  —  modify  the  fourth  sentence  as  follows, 
beginning  with  the  third  line:  "...-not  do  so  centrally  and 

believe  there  is  no  need  to  do  so  because  of  the  existing  system 
of  delegations  and  controls  noted  above." 

Page  3  -  CONCLUSIONS  —  replace  the  last  two  sentences  as  follows: 

"They  centrally  review  grant  proposals  for  administrative  matters, 
adherence  to  university  policy,  and  use  the  academic  governance 
structure  to  ensure  adherence  to  appropriate  scientific-technical 
performance  standards  and  consistency  with  departmental  and  school 
academic  objectives. 

In  most  (all?)  of  the  institutions  with  which  we  communicated, 
formal  evidence  of  the  performance  of  these  reviews  is  required  in 
the  form  of  signatures  of  appropriate  academic  officers  (academic 
department  head  and  dean)  on  a  proposal  validation  document, 
before  the  central  grants  and  contracts  office  will  act  on  it. 

[a  similar  review  process  is  employed  prior  to  the  acceptance  of 
awards  if  changes  are  made  in  either  the  nature  or  direction  of 
the  effort  as  proposed,  or  in  the  financial  or  administrative 
arrangements . ] 

Institutions  with  which  we  communicated  indicated  the  presence  of 
additional  safeguards  for  quality  of  performance:  juried 

publication  of  research  results;  juried  review  of  graduate 
students'  theses  and  dissertations  (which  frequently  are  based  on 
grant-supported  work  and  form  partial  documentation  of  results); 
the  need  to  satisfy  sponsor  scientific-technical  monitors  as  to 
the  adequacy  and  sufficiency  of  the  work,  both  to  be  discharged  as 
to  performance  and  to  insure  eligibility  for  continuing/future 
support;  and  the  requirement  for  technical  competence  reviews  as  a 
condition  of  advancement/promotion  in  the  employing  university’s 
academic  staff ." 

Thank  you  for  allowing  the  opportunity  for  comment.  I  hope  the 
foregoing  is  helpful  to  the  audit  staff  in  acquiring  a  better 
understanding  of  the  nature  of  academic  accountability  governance  as 


84 


APPENDIX  V 


APPENDIX  V 


Osmund  T.  Fundingsland 

United  States  General  Accounting  Office 

July  6,  1981 

Page  Three 


practised  in  this  and  other  colleges  and  universities  with  which 
acquainted.  I  would  also  hope  that  at  least  the  general  thrust  of 
suggested  changes  to  the  draft  might  find  their  way  into  the 
report.  It  is  a  complex  and  timely  topic. 

If  further  information  is  needed,  please  call. 

Sincerely, 

i * 

Earl  G.  L.  Cilley 
Director 


EC:  dp 

cc:  Donald  Kennedy 


I  am 
these 
final 
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GAO  RESPONSE  TO  STANFORD  COMMENTS 

Stanford  University's  Sponsored  Projects  Office  disagreed 
with  our  findings  and  conclusions  regarding  monitoring  of  research 
projects  for  scientific  progress.  Stanford  stated  that  although 
it  accepts  the  responsibility  for  such  monitoring,  as  both  NSP 
and  NIH  grant  policy  manuals  require,  Stanford  in  essence  relies 
on  its  researchers  to  monitor  their  own  projects.  This  is  of 
course  "self-policing"  and  does  not  appear  to  comply  with  the  in¬ 
tent  of  the  agencies'  requirements.  Stanford  also  said  that 
"senior  academic  officers"  are  responsible  for  the  overall  conduct 
of  research  programs.  Researchers  told  us  that  no  one  at  the  uni¬ 
versities  monitors  scientific  progress  of  their  research  projects 
( see  p.  39) . 

University  review  for  administrative  matters  does  not  neces¬ 
sarily  mean  review  for  technical  or  scientific  matters.  Research¬ 
ers  told  us  that  universities  do  not  review  their  proposals  for 
scientific  relevance.  We  did  not  interview  university  administra¬ 
tors  on  this  subject. 
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